Build Your Own Middleware Machine

1 (1.0h)

Build Your OwnBuild Your OwnBuild Your OwnBuild Your OwnMiddleware MachineMiddleware MachineMiddleware MachineMiddleware Machine

Simon Haslam

Veriton Limited

2 (1.0h)

Simon Haslam / VeritonSpecialised consultant & Oracle Partner, established for 15 years

Demanding web & call-centre applications

Architecture & development strategy; health-checks; disaster recovery; tuning

Oracle Fusion Middleware(JavaEE, SSO, OAM, OID, clustering)ADF Applications (esp. strategy & admin)

Database & related technologies(Solaris/Linux, load balancers, firewalls, …)

Agenda

1. My Ideal Middleware Machine2. Hardware3. Software & Configuration4. Non-functional Factors5. Case Study6. Summary

4 (1.0h)

Background to this presentation

• Julian Dyke & I came up with ‘RAC-in-a-BOX’ in 2006

– 2 * HP DL365 (8 cores each)

– 1 * MSA1000 SAN (HA controllers)

– 2 * Cisco 3750 switches

• Standard OFA style installation of RAC, config of ASM etc

• Ready to plug into network & power

�

5 (1.0h)

From the ‘Old’ Days to Today

• Was physical hosts with DAS

• Storage arrays for consolidating storage

• Now: most mid-size sites

– Production: physical hosts for production databases

– Dev & Test: virtualised environments very common

• (leaving discussions about public cloud / hybrid models)

�

6 (1.0h)

From now to future

Appliance

PublicCloud

PrivateCloud

DedicatedHardware

for Application

On Premise

Remote

VirtualisedPhysical

Or heavyconsolidation

Appliance

Compute platformStorage platform

�

7 (1.0h)

Trends

• Green IT– Carbon costing– Power costs– SCR

• Convergence– FCoE– iSCSI maturing

• Technology– Speed of commodity hardware– Virtualisation– Networks VLANs: now traffic management, LACP etc.– 10GbE more common– Not the bandwidth/latency of InfiniBand but may be good enough

for most, certainly mid-size, sites

�

8 (1.0h)

Exalogic is BIG!• Even ¼ rack Exalogic is:

– 8 compute nodes @ 12 = 96 cores– WebLogic Suite list price = £1.4M

• Even if ½ processor & ½ NUP = £870k

– Add: WLS Management Pack, Cloud Management Pack..?– List price of hardware $475k (~ £320k)

• Many of my customers only have two or three big web/call centre apps:– modern processors => production can run on 2 compute nodes (for HA)– with virtualisation 1 compute nodes (12 cores) can run multiple test envs– many still have Forms & Reports – expansion of Java EE apps likely over

next 5 years but not from Day 1– Web app to database traffic is modest

• Nothing wrong with Exalogic… it is just very BIG

�

Oracle OpenWorld 2011: announcement 1/8 rack Exalogic(4 x 12 core 96GB compute nodes, same NAS and IB)

9 (1.0h)

My ideal ♥ Middleware Machine!• Hardware

– Compute: fast processors & ‘plenty’ of memory

– Stateless compute nodes

– Networks for public, private, management

– Power management

– Totally managed remotely

– (note: I’ve not mentioned storage)

• Software– WebLogic

– Web Tier

– Load Balancing

– Management

– Layered products (old and new)

�

10 (1.0h)

Confession: I am a blade convert!

�

Agenda

1. My Ideal Middleware Machine2. Hardware3. Software & Configuration4. Non-functional Factors5. Case Study6. Summary

12 (1.0h)

Ingredients: 1. Compute�

13 (1.0h)

Ingredients: 2. Power

• Single or 3 phase

• Note: health and safety implications

�

14 (1.0h)

Ingredients: 3. Storage�

15 (1.0h)

Ingredients: 4. Network

• Core switches

• Load balancers

�

16 (1.0h)

Bake & Leave to Cool �

17 (1.0h)

Traditional Rackmount vs Blade?

• Power consumptionhttp://www.veriton.co.uk/roller/fmw/entry/...hp_blade_vs_rack_servers

• Costs

• Note some chassis-wide costs:

– Converged networking

– Administration modules

• Vendor lock-in

�

18 (1.0h)

Size of errorbars unknown!

Changesfrom adding fans/power supplies

�

19 (1.0h)

Exalogic1/8th rack 4n (inc storage)

2.4-4.3kW*

* Exalogic “typical” to “max” (from Oracle datasheet 14-NOV-11) – rounded to 2 sig. fig.

Exalogic1/4 rack 8n (inc storage)5.3-7.2kW*

Exalogic1/2 rack 16n: 8-11kW 1 rack 32n: 13-18kW

Note: Exalogichas 96GB RAMper node cf 24GB,SSD boot disksand slightly faster procs(3.0GHz cf 2.8GHz)

�

20 (1.0h)

Compute Node Storage

• What would you put on a server if you had shared storage?– Operating system files?

– Swap space

– Crash dumps

– Application log files

– Transaction logs?!

• Do you need SSD for this?

• As soon as you add storage to a server you make it stateful

�

21 (1.0h)

Hardware Summary

• Very fast processors

• No local storage; boot from SAN

• Shared 10GbE networking (max 20GbE per blade)

• Centralised hardware management

• Centralised power management

�

Agenda

1. My Ideal Middleware Machine2. Hardware3. Software & Configuration4. Case Study 5. Non-functional Factors6. Summary

23 (1.0h)

Provisioning

• Stateless blades (hardware)

– No disk (or SDcard!)

– No MAC addresses

– No WWNs

– No Lights-Out Management addresses

= ~virtualisation of the hardware

�

24 (1.0h)

Hardware/OS Configuration

• Pre-assign IP addresses

– Management

– Public Prod

– Test

– Private

– Cluster

�

25 (1.0h)

Software Installation - today

• Centralised Deployment

– Scripted (e.g. WLST & response files)

– Enterprise Manager – Deployment Pack (still some work to be done)

�

26 (1.0h)

Virtualisation - notes

• VMs – build from kickstart/jumpstart etc

• There’s a lot to learn from AWS

• Oracle Virtual Assembly Builder is a very interesting product for the future

– VM template libraries

• OVAB probably the future if you’re on OVM

• OVAB apparently underpinning Java Service of Oracle Public Cloud

�

27 (1.0h)

Middleware Installation• Shared middleware homes

– Pros and Cons

– Great for support ☺

• E.g. Exalogic EDG

�

Agenda

1. My Ideal Middleware Machine2. Hardware3. Software & Configuration4. Case Study5. Non-functional Factors6. Summary

29 (1.0h)

Case Study

• Mid-sized UK customer

• ~800 agents in 4 call centres + web site

• Previously 10g iAS on 6/8 2U rack servers (production, primary site)

• Migration to WebLogic Suite 11g (plus OID 11g & Reports 11g)

• Moving from Red Hat to Oracle Linux 5.6*

• Virtualised

• Centralised Oracle homes and management

�

30 (1.0h)

How did it go?

• Power was a challenge– you need to size for full capacity

• Cooling– fortunately able to free up space, but could have been

tricky due to heat density

• Boot from SAN– Not really a problem – much more common these days– I suspect (I) will get blamed for anything bad

• Networking– Forced through a rethink of some of the VLANs and

subnets– We did have some problems with LACP/trunking

�

31 (1.0h)

Now?

• A project appeared from blue, sneaked in and went live on the blade system before Oracle project out of testing!

• Chassis will probably be full within a year

�

Agenda

1. My Ideal Middleware Machine2. Hardware3. Software & Configuration4. Case Study5. Non-functional Factors6. Summary

33 (1.0h)

Remember…Licences

and support

Hardware

�

34 (1.0h)

Politics

• Multiple teams are often responsible for support:

– Application Development

– Oracle Administration

– System (e.g. Unix) Administration

– Storage Management

– Data Centre Management

�

35 (1.0h)

Real-World Example in 2011

• Oracle team had bought lots of SOA Suite licences (sizing method a bit unknown – partly historical, partly reseller)

• Production hardware still to be purchased• Licences only efficiently “fitted” on certain combinations of hardware

at primary and DR site• System Administration team different to Oracle/project team• Cost of Oracle licences > 15x cost of hardware• Cost centre for hardware/OS/VMware seemed to be different from cost

centre for Oracle => smallest amount of hardware bought*

� Oracle perpetual licences plus Support/Updates unused

• This is not unusual behaviour!

* I do have some sympathy here – the licences should have been right-sized in the first place

�

36 (1.0h)

Real-World Example: What if they had chosen Exalogic?

• 2 Exalogic ¼ racks (production and DR, would have probably had to run dev/test on DR)

• Presumably Oracle licences would be bought at same time so match

�One decision to make

�1 Purchase Order

�Choice cannot be ‘unbalanced’

�

37 (1.0h)

Exalogic vs Build-Your-Own

©2010 Markus Eisele

�

38 (1.0h)

Exalogic

• Comes assembled & cabled up from Oracle• All decisions have been taken for you – you can only

choose the size• Oracle love you & you’ll probably get a better deal on your

licences• Oracle remote diagnostics & single point of support• You’ll get all the best Middleware features & super

performance, especially if you have Exadata too• OVAB etc means it may be more “plug & go”• Oracle “Red Stack Administrator” is King/Queen!• But:

– May have to buy more hardware than you need up-front– You’ll need at least 2 & possibly 3 – all are same (top) spec– Your admins may not be used to Sun hardware, LOM etc– (Vendor lock-in)

�

39 (1.0h)

Build Your Own (Blade)

• Flexibility to run other stuff on same infrastructure• Very neat cabling• Can buy the fastest processors and migrate slower ones to

test envs• Great automation options• Hardware will be consistent with other non-Oracle stuff• Fits in with virtualised compute layer, storage layer,

network layer => very clean architecture• But:

– You’ll probably need to hire a consultant to set up to get this degree of automation (costly)

– You may have internal bickering over storage, network, boot fromSAN, spec of servers etc

– Oracle hold back best features for Exalogic– With blades you will still have vendor lock-in

�

Agenda

1. My Ideal Middleware Machine2. Hardware3. Software & Configuration4. Case Study5. Non-functional Factors6. Summary

41 (1.0h)

• Exalogic is bigger than many organisations need• Reasons to buy Exalogic may be as much political

as technical• Modern alternative ways to build a “middleware

machine” include stateless compute nodes booting directly from centralised storage

• If building your own - careful configuration required with scripted management – don’t reinvent the wheel. Standardise!

42 (1.0h)

Thanks for listening!

Twitter: @simon_haslam

Oracle Fusion Middleware Admin blog:

http://simonhaslam.co.uk