Upload
paul-gallagher
View
8.676
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Presentation delivered at the Singapore Ruby Brigade meetup 6-Jan-2010 (at hackerspace.sg). Discusses BI and DW in the Rails context, and test drives ActiveWarehouse and ActiveWarehouse/ETL with a "Cupcakes Inc" sample application.
Citation preview
NB: This presentation was delivered at the Singapore Ruby Brigade meetup 6-Jan-2010 (at hackerspace.sg)
BI & DW for Ruby/Rails
“!???”
Why should we care about this enterprisey stuff?
Have you heard a client ask for..– A “dashboard”?– Management reports?– Operational statistics?
..in addition to the actual site?
Or maybe you want to pitch for the dashboard/BI projects
themselves?
..using your rails skills of course
BIBusiness Intelligence
CPMCorporate Performance Mgmt
BPMBusiness Performance Mgmt
B&PBudgeting and Planning
EPMEnterprise Performance Mgmt
DashboardEnterprise Dashboards
BI Basics
No, BI is not (always) an oxymoron
Infrastructure & Systems
BI = Business Feedback & Control Systems
Keeping the doors open
Uptime on the
servers; alerts
Infrastructure & Systems
OperationalManagement
BI = Business Feedback & Control Systems
Keeping the doors open
Optimising in the short termintra-day
Focus on systems in isolation
Need extra call centre
staff on shift?
Daily sales
numbers?
Infrastructure & Systems
OperationalManagement
ExecutiveManagement
BI = Business Feedback & Control Systems
Keeping the doors open
Optimising in the short termintra-day
Focus on systems in isolation
Strategic performancemonthly, quarterly, yearly
Across all systems
Profitability by
product
Utilisation and sales
performance
Infrastructure & Systems
OperationalManagement
ExecutiveManagement
Traditional Rails perspective..
e.g. NewRelic
Custom AR reports
Someone else’s problem (opportunity)
Someone Else’s Problem..
Your Rails Storefront App
Fulfillment(maybe a third party)
To report on sales
fulfillment..
AR/AP/GL
To report on revenue and profitability..
To report on sales revenue,
actuals and forecast..
And don’t forgetall those other systems..
CRM MRP
FA
Who is “Someone Else”?
The gigaohm network: “5 Free Business Intelligence Crunchers for Your 2010 Arsenal”
ODS
ETLYour Rails App
Other TransactionalSystems
Data Sources
DBoR, relational reporting
BI & DW
A copy of transaction data specifically structured for
query and analysis
Extract – Transform – Load
Or, Extract – Load – Transform
Or, Transform – Extract – Load
(depending on the technology)
“cubes”
Sales = $22 Customer ID Product ID Date ID …
Customer dimension Date dimension
Product dimension
Fact categorisation“Fact”
MOLAP, ROLAP, HOLAP
MOLAP: proprietary format to optimize for analytical queries
ROLAP: use relational database to mimic multi-dimensionality
HOLAP: hybrid. Drive analytics from MOLAP, drill down to relational
Star schema Snowflake
Why?? What’s wrong with..
select a.name, sum(b.amount) from products ajoin order_items b on a.id =b.product_id group byproduct_id
Product.sum(:amount, :include=> :orders, :group =>‘product_id’)
•Every question needs it’s own query
•Can’t predict all the questions in advance
•Un-scalable grunt work
ActiveWarehouse ActiveWarehouse-ETL
ActiveWarehouse
Rails plugin by Anthony EdenROLAP solution based on ActiveRecordFeatures
– Generators for Facts, Dimensions, Cubes and Bridges
– Supports calculated fields– View helpers for reports with drill down
ActiveWarehouse-ETL
Rails gem/plugin by Anthony EdenDSL for extract – transform – loadSource/sink: file, db, xml, .. (extensible)Features
– Pre/post processors– Transformations
The Cupcakes Store
Use Activewarehouse-etl to load seed data from csv to app db (mysql)
1
The Cupcakes BI Dashboard
2Use Activewarehouse-etl to load dimension and fact data to the warehouse(mysql to mysql)
3Use Activewarehouse to build a simple analytical “dashboard” and reporting tool
Follow the documentation at http://github.com/tardate/cupcakesinc to see how this works (and try it yourself)
Product listing at Cupcakes Inc..
Customer listing at Cupcakes Inc..
Order listing at Cupcakes Inc..
Order detail at Cupcakes Inc..
Sales By Product AW Report
Sales By Product (drill to 2009)
Reasons to be Cheerful..
Language
ETL processing, cube rules etc typically use custom languages (often archaic and limited)
BISuites
It’s … ruby!
UI Customisation and Presentation Integration
Web delivery typically very constrained.
Often rely on strong integration with office software (Excel). Leads to “custom application development in Excel” syndrome.
BISuites
It’s … ActionPack!
Google maps mashups, social graph links. .. you get full UI control, as long as you have the development budget.
Speed of development
Basic deployments can be very fast.
But UI inflexibility can lead to either lots of time wasted trying to shoe-horn, or need to “reset customer expectations”
BISuites
It’s … Ruby & Rails. Say no more ;-)
TCO
Top-tier suites can come with a hefty $ tag. And prices are going up..
But some analysts are predicting 2010 to be the year BI gets FLOSS momentum (see gigaohm review of 5 well established alternatives)
BISuites
It’s … Ruby & Rails. Say no more ;-)
Trade-in software license costs for more development.
Caveats..
Native MOLAP
Generally good support for database MOLAP features.
Can be platform specific though – e.g. Microsoft MDX, SQL Server Analytical Services
BISuites
A gap. No real support currently available.
ActiveWarehouse uses relational model to “fake” MOLAP (ROLAP)
Performance
Generally, all established analytical engines (and backing databases) have great performance track record. Huge scalability (millions of rows)
BISuites
Unproven. ActiveWarehouse/ETL does not have many (public) proof points.
Given that it is tied to AR performance, expect scalability could be an issue.
Take-aways~ActiveWarehouse
It’s an impressive codebase. When you get it working, it works well.. but
– Virtually no documentation!– No contemporary examples– Not under very active development– A “textbook” data warehouse implementation.
May or may not be exactly what you want..
Remember: – data is batched. Not realtime.– Rails 2.x : install the plugin (gem is 1.x)
3
Take-aways~ ActiveWarehouse-ETL
Neat tool. In addition to feeding AW:– Generate and load seed/test data– Move data between systems
But again,– Poor documentation– When it fails, can do so silently (makes
sure filename paths are delimited correctly for your platform!)
2
Take-aways~ BI on Rails Solutions
Plain AR– just avoid the rabbit hole
AR + ETL– get all the data you need in one place
AW+ETL– traditional ROLAP, make Rails the focus of the BI effort
Go the BI suite route– When you need to adapt to many transactional systems
at scale, and customer has the $$ – (Rails remains just for transactional apps)
Or… (discussion point;-)
1
Thank you!
Questions?
0
Some References
ActiveWarehouse: http://github.com/aeden/activewarehouse ActiveWarehouse-ETL: http://github.com/aeden/activewarehouse-etl Cupcakes Inc sample site(s): http://github.com/tardate/cupcakesinc Singapore Ruby Brigade (SRB): http://groups.google.com/group/singapore-rb