Notice and Disclaimer Warning - YOW! Conferences and ...yowconference.com.au/slides/yowwest2014/Thomas-FAST...Real-time DB + Batch DB (Twitter Storm) 2014-07-02 13 Testing is still

2014-07-02

1

© 2014 Bedarra Research Labs. All rights reserved.

FAST Software = FAST HW + Simplicity

Dave Thomas www.davethomas.ne

Bedarra Research Labs, YOW! Conferences,

Carleton University, NITCA Fellow, Lodestone Foundation, Depth First

Notice and Disclaimer

Warning This talk is intended for a mature open minded

technical audience with a good sense of humor. It contains future looking speculative

opinions not endorsed by YOW!

2014-07-02

2

Outline

Software enabled by HW

FAST Hardware 2014 – 2020+

Faster and Cheaper?

Simplicity – Software Sympathy*

•  Simplicity in Code Today

•  Simplicity in Architecture – IT 4.0

•  Functional Thinking

* Mechanical Sympathy – Martin Thompson


Software Improvements enabled by Hardware

© 2012 Bedarra Research Labs

•  Procedural Abstraction – 1960s

•  Multiple Processes and Pipes – 1970s

•  Stacks and Nested Scipres – 1970s

•  Object Abstraction 1980s

•  Garbage Collection 1990s

•  Faster CPUs by Superscalar 1990s

•  Large Disks by Caches and Faster Rotation 1980s..

•  Distribution by Faster Networks 1990s ..

•  Faster Storage via Flash/SSD 2000s

•  Cloud Computing and Mobile 2000s

•  Continuous Delivery 2010…

•  Big Data 2010 …

•  FP, Micro Service Architectures 2010 ..

2014-07-02

3

Lost on BigData


2014-07-02

4



Faster to Send to another core than store in memory!

2014-07-02

5

1000 cores in 10 chips!


Massive Scale Not Far Away!


2014-07-02

6


Generals talking to defense contractors “What can we add to the new stealth fighters that will cost $1B but not add any weight?”

Contractors “We have it! It is called and it is never done”

Top Secret

Faster and Cheaper?

How can we reduce the software cycle time?

How can we reduce execution cost?

©2009 Bedarra Research Labs. All rights reserved.

Needs Solution

Development

2014-07-02

7

Let the Hardware Do The Work! $100,000 buys a computer 1 TB RAM with 40 TB disk and 8 -16

cores!

$500 buys 1000 4 GB cpus on Amazon for 1 hour!

•  Automated Build and Test

•  All interesting data is in memory!

•  Inexpensive Data Conversion/Translation

•  Data Compression and Encryption is “free” on multi-core

•  Data Base is an oxymoron

•  Speed and Memory enable Simpler Algorithms

•  Enable End User Computing e.g. Excel and R versus C#, Java


Ship Less Code!


KLOCS Kill! •  The more code you ship the more code you need test

and fix! Dependencies Strangle!! •  Increase Integration Testing •  Increase Build and Deploy Time •  Often pull in Much than required

Abstractions Bloat! •  Bloat Costs

•  Development Time •  Execution Time and Space

•  Frameworks inject dependencies into your code!

2014-07-02

8

Living in a Legacy – Big Ball of Mud Where does MUD come from ...

Use OO!

Legacy is code where there are no tests - MF

Perhaps we should reconsider emergent design!

OO Domain Models Considered Harmful

With very few exceptions most software consists of data and functions which operate on it.

1.  OO Domain Models often mandate an ORM which are known performance bottlenecks. Record->Object conversion, GC

2.  OO Domain Models introduce a lot of navigattion code which is unnecessary

3.  OO Domain Models often miss many to many relationships in data model


2014-07-02

9

Use Less Objects and Less Code !

Object Refactoring harder than Changing Data and Functions


Table Driven Programming

Rules Decision Table Calculation Spreadsheet Data Validation Domain and Range Table Mapping Lookup Table Flow Data, Work Flow, Message Events, Matches State Table Process, Reports Input-Output Table Acceptance Criteria BDD Domain Models Entity-Attribute Dictionary

Enable Customer Self Service Atom feeds versus APIs Query versus custom Expose Scriptable Services Self Described Data

Table Driven Programming A picture is 1000 words, a table 200 and a diagram 50

Advantages

•  Easily understood by Business, BA, Dev and QA

•  Easy to create, refactor and extend using Excel

•  Modularity through structured tables

•  Consistency /Completeness Checking

•  Easy to version and Diff

•  Efficient Automated Data Driven implementation

•  Data Driven means changes can be “hot deployed” to a running application

Applications Insurance, Banking, Taxation, Healthcare, ATC, Real-time...


2014-07-02

10

Reduce Integration Time and $$$

Avoid Slow and Complex APIs •  Leverage stable physical data formats – pages; logs

•  Make R/O SQL replicas for fast reporting

•  Put ATOM/RSS feeds on our legacy/partner systems – journal files, events …

•  REST and JSONify your services

•  Use ODBC as a simple interface to complex server systems

•  Use a simple MashUp tool against to deliver a integrated application view


Script to Save Time and $$$ •  More and more applications are disposable at least in part –

make it work and get it out there, scale it later if you need to ( e.g Facebook …)

•  Use Ruby (IronRuby, JRuby) , Python(Jython, IronPython) , PHP, Groovy, JavaScript, Clojure, Scala Scalding… to get development productivity

•  Leverage SAAS services

•  Leverage core internal and external services via REST/ODBC


2014-07-02

11

Why Binary Data Formats are Important

1.  Textual data formats such as XML, JSON, CSV are expensive to parse

2.  Textual data formats are bulky to store

3.  A trivial print library will allow easy inspection of binary logs

4.  Make sure your format has a version id for every format (aka schema change)



ind

A Simple Matter of Programming!?

NoSQL - Why query when you can write a program?

2014-07-02

12

NoSQL – Good! News Bad! News

Good News !

•  Schemaless

•  JSON everywhere

•  Query is just programming

Bad News!

•  Schema adds value (especially over time)

•  JSON parsing (just like relational blobs)

•  Query is just parsing and programming


Batch, RDB + HDB, Event Source, Lambda Architecture


Real-time DB + Historical DB Real-time DB + Batch DB (Twitter Storm)

2014-07-02

13

Testing is still 30 – 50% of Development

Test Soon – TDD, Inspections, Acceptance Tests

It all starts with acceptance criteria (including ilities)

SBE/Cucumber Good|Bad News?

Robust Testing Requires Independent Models

•  QuickCheck is currently the best examplar •  Generates random tests from a spec •  Collapses Nonessentials Test Results •  RIAK, Level DB experience


We all know that testing costs a lot and takes time, especially when working in a changing complex environment. Lets not bother!

What it takes 1.  Modular micro service architecture §  instance deployment and tear down §  loosely data coupling

2.  Simple Functionality Deployment

3.  Stringent Monitoring for Deviant Behavior

4.  Applications – Telecom, Finance, Web, Sensors …


Code-Deploy-Monitor – Testing Considered Harmful!

2014-07-02

14

Simplicity! - The Road Not Taken?


Complexity Simplicity

The More Things Change? … IT Stays the Same!


Create – Read –Update – Delete (CRUD) Business Analytics, Mobile, Cloud Computing, Multi-player Games …

2014-07-02

15

IT is still CRUD, Workflow and Data


Input and Outputs - Forms, Reports and Data Transformations

Workflow – Form Flow and Transformation (Boxes and Arrows)

Event/Actions – Event-State Tables

Rules/Actions – Decision Tables (Rule Engines)

Complex Calculations – Constraints/Dataflow (Spreadsheets)

Data and Relationships – Data Models (ER=OMT)

IT 4.0

Challenges ●  IT 1.0 - 3.0 Rigidity and Lost Opportunity

Opportunities ●  Simplfied Flow - Value Driven

●  Accelerated Delivery

●  Data Driven - Query Oriented

●  New Modularity - Microservices and Actors

●  Self Service - End User Ultimate Pairs

2014-07-02

16

Service-Oriented Computing Infrastructure: Cloud -The Software Enabler

The Emergence of A Simpler Application Infrastructure §  Examples - On Demand, Software As A Service such as Amazon S3, EC2,

SimpleDB, Google App Engine, Sales Force … §  Simpler limited “thin” service API (< 50 ) closer to underlying platform which

provides support for scalable, distributed, secure computing §  Independence on mainstream vendor Underware and Middleware §  Google Linux, VMware Virtual Machine, MS Azure Hypervisor V. Google “Pinnacle”

Application Development Benefits §  Small Service API (thin to none class library & frameworks) §  Limited Choice Reduces Decisions and Support §  Leverages Other Apps through Services §  Total App Responsibility from envisioning to production

i.e. App Team caries the beeper

Evolution of IT Application Architecture

App Tier Mid Tier Data Tier

My Project My Project

My Schema

App Tier SOC Tier Data Tier

My App My

App

My Apps

SimpleDB, S3 GFS, Bigtable SQL Services

Cloud Services

Hosted or Desktop Apps

My Project

2014-07-02

17


What Is A Service? A service provide an interface to a specific subset of functionality/

data in an enterprise. § A service is provided by an API realized by one or more value

based interfaces, ideally independent of a specific technology. §  The external service interface) definition and its SLA are the total

knowledge required to use the service. §  SLAs are verified by monitoring SLA at execution §  Services are designed to be composed into larger services or

Business Apps. §  Services composition consists of simple data flow “pipes and filters”

and more complex flows by orchestrated actors. § Versioned Services can be concurrently deployed to enable new

and older apps. §  Services are not frameworks; they are components with value only

interfaces. §  Services are realized as components with consumable APIs. § One or more services can often be supported by a service team.

Components versus Frameworks

•  Frameworks enable the development of components by leveraging OO principles of encapsulation and reuse.

•  Frameworks require the component developer/framework user to understand the insides of the frameworks.

•  Frameworks “inject dependencies” on the framework into the application.

•  Frameworks often provide many ways of doing things, including ways to specialize the framework.

•  Components provide only a small number of value based interfaces .

•  Components are ideally black box, where as frameworks are white box out of necessity.


2014-07-02

18


The Business Value Proposition

Business Apps can be more easily configured from Services §  Large Monolithic Systems are decomposed into simpler services

which enable §  Services and their SLA published in a web catalog § One or more services can be composed into an App with a tailored

interface for a given market § App delivery can be supported by simpler less technical end user

tooling such as Wikis, Visual Connections § Apps can combine services from insurance and banking etc. §  Bus Apps = Services (Features) + Services (Components)

Bus App Development


Do It Ourselves Programming –The Empowerment Business Driven Development

§  Enterprise Mashups – The Real SOA? §  Applications Assembled from Feeds and Services

Lively Fabrik

IBM QEDWiki Yahoo Pipes

Dabble DB

Google Mashup Editor

2014-07-02

19


Technical Value Proposition

Services increase modularity, reduce coupling, increase technology and delivery choices. §  Services reduce large monolithic applications §  Services are loosely coupled hence can be incrementally be

developed; easier to distribute, provision and monitor. §  Services expose the tangible business architecture versus the

internal technical applications architecture §  Services are realized as components with consumable APIs. § One or more services and be owned and implemented by a team. §  internal technical applications architecture §  Services enable parallel Business App development and service

definition and development (feature and component teams) §  Services are realized as components with consumable APIs. §  Services easier to deploy and test. § One or more services and be owned and implemented by a team.

Loose Coupling – Let’s Hope It Sticks This Time

Data Flow – Data Flow Computing … Maxor FPGA DF

Structured Analysis and Design (SADT)

Flow Base Programming – J. Paul Morrison

Unix Pipes and Filters

Hewitt Actors

Spreadsheets

Visual Programming - Labview

Erlang

Messaging

Data Driven


2014-07-02

20

Enabling Loose Coupling

•  All APIs are value based and where possible stateless

•  Isolation of services in separate processes/machines

•  Simple Pipes and Filters when possible

•  RSS/ATOM feeds from events/updates/logs

•  Query interface such as ODBC

•  Occasionally Disconnected – replication and sync; event source..

•  Simple efficient implementations using co-routines..

•  Orchestration/Composition using Scripting Process •  Service Bus, Messaging.. ZeroMq, Rabbit •  Node.js e.g. Azure Mobile Services •  Erlang, Actors, Scala Akka

•  FP thinking encourages value orientation and composition



Service Architecture and Design

2014-07-02

21

Future of Programming is Query! Data Driven Functional Transformations

Historical Views of Programs

•  Program = Procedures + Data Structures

•  Program = Process + Data Flows

•  Program = Expressions + Constraints

•  Program = Logic + Control

•  Program = Messages + Objects/Actors

•  Program = Pure Functions + Actions

•  Program = Queries + Actions!?


Collection Oriented Programming

•  Programs written with a functional style of map, filter … are • More concise and easier to read. • Reduce side effects • More amendable to optimization

•  Examples •  Underscore.js; CoffeeSript, Elm.. •  LINQ, Rx, Reative Streams, •  Map Reduce •  Clojure, Scala, Haskell, Erlang, Ocaml, J, Java 8+, C# 4.0 …


2014-07-02

22

Go Forth and Simplify!

Thanks!


Documents

Notice and Disclaimer Warning - YOW! Conferences and ...yowconference.com.au/slides/yowwest2014/Thomas-FAST...Real-time DB + Batch DB (Twitter Storm) 2014-07-02 13 Testing is still