NUMBERATE 8.0Rhino
Technology Overview
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
The Numberate Rapid Warehouse Solution is a data integration solution that maximises the cost efficiency of data oriented business intelligence environments.
It builds on over a 15 Years of experience delivering these solutions to a range of corporate and government clients who needed to leverage business growth from their organisations underlying data assets.
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Numberate Rapid Warehouse Solution
The Numberate Rapid Warehouse Solution differs from other solutions because of its flexibility, its non-proprietary architecture and its speed of implementation.
Based around the concept of business rules, it incorporates a high degree of reusability allowing customers to leverage and extend their existing investments in data warehousing and business intelligence.
Numberate Rapid Warehouse Solution
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
The Numberate Rapid Warehouse Solution is the right solution to meet the needs of your business for going forward. You need to give your IT professionals tools that;
1. Allow them to focus on your business rather than our technology.
2. Provide options and alternatives rather than mandating yet another upgrade or add on.
3. Deliver in weeks rather than months.
4. Are driven by your business rules and processes rather than driving your business broke.
Data Warehousing & Business Intelligence Solutions
Numberate Rapid Warehouse Solution
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Solution Requirements: Business
THE CHALLENGE
Business users don’t understand their data
Information is power “Its distribution is threatening”
New information, or new views, changes requirements
Timeliness is everything
“Yesterday’s issues are meaningless today.”
THE NEEDS
• Involve business to
– Confirm solutions
– Change their mind
• Rapid development through automation
• No penalties for rework
• Return control of data and distribution to the business
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Solution Requirements: ETL
THE CHALLENGE
Data from disparate systems doesn’t match
Cross systems business rules difficult to model
Dirty data blows out exception handling
Users change their mind
Upstream systems constantly change
THE NEEDS
Capture business rules once
Allow users to prototype inside the ETL process
Automate the process
• Maintaining the rules
• Defining the transforms
• Defining the target environment
Provide capacity for rework
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Solution Requirements: Project
THE CHALLENGE
Development program midstream
Fixed price contract
Existing tool sets
Existing development partners
Cost constrained project
Project schedule pressure
THE NEEDS
No upfront investment
Fixed price
Reduce development schedules
Reduce cost and time for work packages
Provide alternate approaches
Data Warehousing & Business Intelligence SolutionsDecember 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
A different approach
The Numberate Rapid Warehouse Solution automates the process of building and maintaining a warehouse
→ Open, off the shelf, commodity technologies.
→Solution prototyped before decision.
→ Performance benchmarked before full implementation
→ Independent of target technology
Value Proposition
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
A different approach to building a data warehouse
Solution Space
Warehouse Development Environment Rapid transformations Prototyping Maintenance
Warehouse Integration Environment All major tools and RDBMS’s
Warehouse Enhancements Storage Front end Backup archive and recovery
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Front EndPower Users
Numberate Architecture
Standard Reporting Tools
ODBC
Detailed Data Cache or Store
Lo
gic
al
Vie
ws
RD
BM
S A
PI
RDBMS ODBC
RDBMSRepository
or DataWarehouse
ETL Environment
Operational Systems
Operational Systems
Operational Systems
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Numberate Matrix
PhysicalDataElements
AdditionalLoad Code
Raw Data
Raw Data
LoadExecuteData
Code Generation Engine
User Data Request
Result Set
End User
LogicalDatabaseDefinition
User Query Builder
User Data
Request
Result
Metadata Manager
User ElementSecurity
Intersection
Numberate Matrix
Numberate Matrix Manager
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Technology Features
Platform Transparency
Uses inexpensive, standard components: hardware, software Disk
Uses any storage devices orthogonally disk, tape, CD ROM,
Independent of front end tools and back end data stores
Horizontally scalable on any platform
Does not require identical hardware
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Value Proposition
Single metadata repository
Extract and transform data
Define warehouse environment
Define third party environments
Maintain, schedule and operate warehouse
Scalable storage solution
Terabyte enabled
Uses standard server / disk environments
Proprietary partitioning technology
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Metadata- Magic or Myth
Multiple standards
Multiple repositories
Residing in multiple separate systems
Addresses only a partial view of the environment
Not integrated with operational environments
Usually managed as a task additional to the implementation
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Metadata
SOURCES
Numberate metadata generates efficient, error free code for the entire warehouse environment
NumberateMetadata
Business Rules
User Regts
Manual Inputs
Existingwarehouse
Datamart
Sourcedatabase
ExistingMetadata
USES
NumberateMetadata
ExtractionRoutines
DefineStaging
area
Transformationprograms
Definewarehouse
tables
Definetableloads
Generatetests
Loadfrontendmetadata
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Numberate Metadata – It’s Alive..
Active Automated Accurate Essential Self documenting End to end Reusable Extensible Pervasive
One and done approach to build a warehouse
The tool manages the project work
Reduces effort, time and cost
Maintain the warehouse through the metadata
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Numberate Kernel
Numberate Unplugged
Code Repository
RWSAdmin Interface
RWSUser Interface
3rd Party Metadata Interfaces
Metadata Repository
Metadata activates program objects
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Staging & Test area
Source transactional data is partitioned for parallel processing.
Partitioned at table level.
Can be repartitioned for performance tuning with minimal rework.
Parallel operations come from standard systems multi threading.
SummariesAnd Derived
Data
Standard Operating System
Java Environment
Numberate Environment
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Unstrained Integration
By architecture Hub and spoke Enterprise Focused Data Marts Normalised / Schemas
ETL Direct database load Parallel ETL processes Auditable Low maintenance costs
Prototyping/ Front end Derived from metadata Low cost Rapid development times
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Integration Services
Numberate Integration Layer
DW
Legacy Repositories
New Datamarts Emerging Technologies
Data & Event Bus
Front End Environments
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
TransformationEngine(s)
TransformationEngine(s)
Typical Transformation Architecture
Target logical models cannot be validated by users without the data
Data quality makes inferred relationships
difficult to code
Transformation Engines are single stream, linear processes
Disparate systems cannot sustain desired relationships
Data Architectures and models may need to be radically altered:PerformanceData QualityBusiness needs
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Iterate the Stream
Traditional Transformation Process
OracleSourceSystem
OracleSourceSystem
Teradata Target System
Teradata Target System
Coboldata
extractionprocess
Coboldata
extractionprocess
Fastloador
Multiload
Fastloador
Multiload
TeradataSQL
Programs
TeradataSQL
Programs
Transformation processes occur at all points
• Difficult to plan• Hard to code manually• Very hard to change or rework
Single stream approach• Limits transformation models• Requires many iterations.
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Traditional Transformation Process
1. Use the warehouse for ETL
2. Co-ordinate many environments
3. Co-ordinate multiple teams
4. High project management load
5. High documentation load
6. Target environment is not independent of ETL stream
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Non Proprietary Architecture
The Numberate Rapid Warehouse Solution leverages the power of modern multi- tasking operating systems and advanced storage solutions to provide scalable data environments that can operate in near real time and provide rapid access to complex
queries in business time.
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
ETL Strengths
ETL has full units of parallelism Multi staged ETL developed as one process
Implemented as different streams
SAS
Utilities
SQL
C / Cobol Sophisticated high speed pattern
matching Matrix Input Loader Generic Data Conversion Module Saves tens of thousands of lines of code
Generated code is error free Metadata driven End to end – including warehouse definition
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Elements of Compatibility
STORAGE
Disk / tape / CD-rom
Appear as RDBMS views
Low cost storage
Suitable for transactional histories
DATA ARCHITECTURE
Normalised data models
Star Schemas
Snowflake Schemas
DATA STORAGE
Mix and match capability
Oracle / DB2 / Teradata/SAS
Native file systems
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Meeting Your Needs
End to end automation reduces time and cost -Provide faster, cheaper work packages
Capture existing sources of metadata-Warehouse work already completed-Definitions in old repositories and data marts
Provide you with more options-Transformation can be done on
• Mainframe• Data Warehouse• Unix
-Move between platforms as run time option
Solution includes a RAD / prototyping environment-Model and confirm business requirements-More accurate reporting specifications
December 2, 2014Numberate Data Warehousing & Business Intelligence Solutions
Recommended