High Availability SOA APP with GlusterFS

What is SOA The Application The Solution How to do it DEMO Time Closing

High Availability SOA APP with GlusterFS

Vladimir Vitkov

2015.08.15 / VarnaConf

High Availability SOA APP with GlusterFS Vladimir Vitkov

In the world of big systems and soa the services need to be highlyavailable and always accessible. If developers forget to implementthis, ops needs to step in and feel the void. This presentation aimsto show you a relatively simple way to achieve this using theclustered filesystem GlusterFS. The main accent is on organisingsynchronisation, replication and disaster recovery.


Who am I

• Vladimir• Sysadmin / DevOPS• FOSS proponent and addopter• Experimentor• Relatively nice guy for a chat*



What is SOA


Let’s put the basis on what SOA is and why it is usefull for the world


The monolith


The monolith approach is usually the first one. It is easier toconstruct as everything is in the same place. There is no need toaccount for delays, communication failures and similar. Of course ithas it’s drawbacks. The piece of code is huge, hard to maintain andscale. Sometimes ugly, but always hard on supporting.


Microservices


The microservices approach usually evolves from the monolith one.The big APPLICATION (yes all capital) gets split at some point intosmaller more contained chunks that do few things (initially). Thefinal incarnation is one service does one and only one thing. Thebenefits are easier maintenance, scaling and supporting the code.Usually this is done with a total rewrite but sometimes it is donewith a state of mind as refactoring code.


SOA


SOA is mostly a pattern in software design in which the componentsof the system are separated from one another and communicate viaa protocol (usually network one). The principles are neutral andindependent of vendor, product or technology. Most often this isachieved with the help of the protocols. Most common ones areSOAP and REST. As such it is in fact microservices architecture buton a conceptual level. Currently large-scale applications are based onthe SOA principles and microservices model.


The Application


Well one of the services only


What is the system

• Large scale Ecommerce product• Quite a few services that interact• Lots of users• Lots of traffic


I am helping run a fairly complex and large application, we havequite some services, lots of interactions, lots of traffic and users andthis needs to always be available and run.


What is the service

• Brand New service• Replacing custom hacks• To be in service in 3 weeks• After several total redesigns


The service i am going to tell you today was brand new. It was bornin 3 weeks. Main purpose of it was to replace some custom hacks inour processess and more fluidly control access to resources. It hasbeen totally redesigned several times. It has the core of it changed atleast 3 times. And finally introduced without anyone noticing (exceptdirectly involved parties).


What is the service (2)

• Speaks REST• Stores some local data• Transforms requests• Pulls data from 3rd party service• Is stateless


This service speaks rest, stores some configuration in local storage.When someone asks it the right question, respoonds after pullingsome 3rd party data, applying transforms and generally hummingalong. The best thing: it was stateless.


Wha the service was not

• Highly available• Redundant• Without central data store• But needs to be• Does not know of changes


Unfortunately not everything is hunky-dory in the SOA land. Theservice was not highly available, not redundant, had no idea ofcentral storage and generally was not ready for prime time. But itneeded to be all this things. So OPS had to rescue it and give itwings.


The Solution


Leave the devs to develop and ops to operate. But join them for asynergy.


Highly Available

• Bring up more instances• Spread them around• Load balance them• Monitor them for health• Recover/replace them


To make the service more highly available do the sane thing. Theservice is stateless so apply the regular tricks. Add moreinstances/server. Put them in a load balancer (either DNS or someother like HAProxy). Of course do not forget to monitor the stuff.Use your best judgement. Monitoring is moot if there are noprocedures to detect, recover or replace failed instances. Doing thisin an automated fashion helps a lot.


Redundant

• High Availability• Monitoring• Recover/replace


Keep that service stateless and Highly available. Redundancy isnecessary for high availability anyways.


No central data store

• Core issue• Create GlusterFS cluster for replica• Reconfigure app to use clustered data store• Monitor it• Recover/replace


The missing central data store become the core issue that prompteddeveloping this approach. The application was aware only of localstorage and porting it to support remote/central storage would havemissed deadlines. So ops had to rescue the situation. The solutionwe thought of included creating a replicated GlusterFS cluster. Ittook care to synchronise and replicate the needed local data to allnodes. After that it was just necessary to reconfigure the applicationto use this local (but in reality replicated) data storage. As arecurring theme do not forget the monitoring. Monitor allcomponents available and monitor them well. Again do not forget toautomate recovery in case of troubles with the nodes.


Become aware of changes

• App does not know about new data• Build update mechanism• Inotify + curl• NFS is not a friend• FUSE is not a friend too


Another problem with this approach was that the application is notmonitoring local data store for changes. To solve this add amechanism to inform it that the data store has changed and shouldbe reread/reconsidered. Devs built a mechanism to do this andexposed it. The detect the changes on the filesystem level andtrigger this mechanism. Inotify and curl fit nicelu into this. Butunfortunately NFS and FUSE do not support inotify events. FUSEmay support them but requires special codding in the FS to have it.


How to do it


Show how it is built


Components

• GlusterFS• nfs-common• inotify-hookable• nice utils around


First step is to install everything necessary. This includes theGlusterFS itself, nfs client, inotify monitoring tool and any utilitiesyou may need


Create GlusterFS cluster

• Start gluster daemon• Assemble cluster

gluster peer probe remote.peer.host

• /var/lib/glusterd/peers/*


Next step is to build the cluster. Building the cluster is basicallyprobing the peers. Do the probing from one node only. No need todo it from the other nodes. Make sure your DNS works properly andcan resolve the names. If you don’t want to mix names/ip addressessyou may have to manually edit gluster peers to use names.


Create volume

• Create replicated volume

gluster volume create vol1 replica 2 \transport tcp peer1:/vol peer2:/vol


Creating the volume is a simple command. This will initialize the socalled bricks that form the GlusterFS fabric. As we are aiming forredundancy and replication we create the volume with replica count2 and add only 2 bricks. If bricks are created on the root partitionyou will need to use "force"option. Take some time to review theGlusterFS tunning options (Resources slide).


Reconfigure

• mount

mount -t nfs localhost:/vol1 /app

• Start the app


We chose to use local nfs mount to enable functioning of the servicein a split brain/down situation. The data is replicated everywhere soit is not important from where we serve. GlusterFS has areconciliation mechanism that can handle split brain situations. Ofcourse chosing NFS (FUSE also won’t help you) leads us to the nextproblem. No inotify events


Update info

• Data added on other node

inotify-hookable \--watch-directories /vol \--on-modify-command curl


It is important to update the applications view of the data store. Wedo this by detecting changes (to the local brick) and triggering theupdate cycle with curl. monitoring the local brick is necessary due toinotify events not being emitted for nfs.


DEMO Time


In this demo we’ll show you the final result. We’ll play with disablingthe update mechanism, so we can demonstrate that content isproperly replicated but the application is not aware of it.


Closing



Q/A?



Contact

• Demo: http://is.gd/gluster_demo• Slides: http://is.gd/gluster_varnaconf• mail: [email protected]• GPG: A162 1211 8ACB 4CC5



Resources

• http://is.gd/gluster_tune• http://is.gd/gluster_quickstart• http://flask.pocoo.org/


Technology

High Availability SOA APP with GlusterFS