Building Stream-Analytic Ecosystems for Enterprises
Approved/External Publishing/Raman Ganesh
Building Stream-Analytic Ecosystems for Enterprises1
AGENDA
• Part I : Ecosystem needs of an Enterprise for Stream Analytics
• Part II : Introduction to Meta-programming
• Part III : Metaprogramming on-top of Apex: Demo
• Part IV : Agility For Analytics In Enterprises
• Part V : How it is different from Apache Beam Model?
2
Ecosystem needs of an Enterprise for Stream Analytics
4
Type systems composition for
Domain Validations
Individuals, Interactions,Platforms,Processes
and Iterations
Ideation to realization latencies
and information exchange
End to end revision management
Domain refactoring
Integration, sandboxing, prototyping Productively
Managing different Semantic Models
Formal Semantics for Domains
5
Denotational Semantics
Operational Semantics
Axiomatic Semantics
And their sub-sets
Metaprogramming
7
A metaprogram is a program that generates other programs or program parts
Code is data and Data is Code
Metaprogramming can be used to move computations from run-time to compile-time, to generate code using compile time computations, and to enable self-modifying code.
Usual types of metaprogramming :1- Interpreter integrated metaprogramming e.g. byte-code rewrite for JVMs
2-Dynamic expression evaluation from specific language ecosystems3-General purpose program transformation
Language-oriented programming
8
Language-oriented programming (LOP) is a style of computer programming in which, rather than solving problems in general-purpose programming languages, the
programmer creates one or more domain-specific languages for the problem first, and solves the problem in those languages
The concept of language-oriented programming takes the approach to capture requirements in the user's terms, and then to try to create an implementation
language as isomorphic as possible to the user's descriptions, so that the mapping between requirements and implementation is as direct as possible.
An Interactive MPS preview of a DSL built ontop of Apex
10
Domain model only concerns user models
Operators are type-safe and formally verifiable with custom type rules
and structured
Comparison of the stock Apex CSVOp example in
Java vs DSL
Denotational semantics
Execution Semantics
Meta-concepts around the demo
Part IV : Agility For Analytics In Enterprises
Building Stream-Analytic Ecosystems for Enterprises11
Domain Driven Analytic ecosystems
12
Facilitating iterations : productively
Process built on a platform
Change latency for rapid analytic requirements
Ideation to realization latencies
Standardized workflows placing
Individuals and Interactions at centre
Common ecosystem problems solved with a meta-programming
platform!
Part V : How it is different from Apache Beam Model?
Building Stream-Analytic Ecosystems for Enterprises13
Beam model vs Metaprogramming approach
14
Beam model : Domain is data flow pipelines
Domain is the actual business domain
Denotationally hides underlying runners towards semantic
encapsulation but exposes Beam semantics externally
Denotationally hides underlying platforms and exposes domain semantics to the highest levels
Does not target analytic eco-system problem but only analytic platform
eco-system
Solves the meta-level problems for an analytic ecosystem and can easily
compose on top of Beam model denotationally