S309137

Migrating Applications from Local to Distributed Caching with Oracle Coherence

Contents Introduction: Oracle Coherence Hands-On Lab ........................................................................................ 3

Base Environment ................................................................................................................................... 3

Quick overview of the application............................................................................................................ 4

Running the application with local cache ................................................................................................. 5

Explore the application. Understand how it works. ................................................................................ 11

Update application to run with distributed cache .................................................................................. 13

Configuration Subsystem ................................................................................................................... 15

Cache Store ....................................................................................................................................... 16

Distributed Cache Worker implementation ........................................................................................ 16

Run application with distributed cache .................................................................................................. 18

Test failover and redundancy ................................................................................................................ 22

Shutdown everything ............................................................................................................................ 22

Introduction: Oracle Coherence Hands-On Lab

In this hands-on lab, you will take an application that loads data from a set of large files, caches the data

in memory, and performs complex queries and updates against the cached data, with changes persisted

to the file. You will then modify this application to cache the data in a coherence data grid and perform

these computations against it, learning the common Coherence API's, configuration files and usage

patterns as you go along. By doing this, you will become familiar with some core coherence concepts

like the configuration sub-system, the read-through and write-through mechanisms for loading and

persisting data to the backend data source, the JMX subsystem, and the use of EntryProcessors for lock-

free concurrent and well performing updates.

For this hands-on lab, we have built a sample application that simulates the access and update of a set

of dictionaries of some outdated (and frankly made up) languages. In the next section, you will get an

overview of the application. The initial version of the application will work against the dictionaries

loaded up in your application's local JVM. We will then walk you through updating the application to

work with the data loaded in a Coherence data grid.

This hands-on lab is focused on Coherence. There is no dependency on a database or an application

server. The knowledge you gain here is directly applicable in use cases where Coherence is used within

an Application Server or fronting a database.

Base Environment The lab machine provides you with the following environment that you will be using in this lab.

Oracle Coherence 3.5 for Java

Eclipse 3.5 Galileo

JRockit 1.6.0

On the desktop, you will find a folder named S309137 Migrating Applications from Local to Distributed

Caching with Oracle Coherence. This folder contains a shortcut to the lab which you will be working

with. The actual lab folder (accessed from the shortcut) contains scripts to assist you with the lab. There

is also a solutions directory in there, which contains the solutions for this lab (for reference when stuck,

etc).

Quick overview of the application

This sample application allows you to access the following information about words in a set of

languages.

description

synonyms

antonyms

daily frequency of use of each word

the year which the word became obsolete

The words and their metadata (as described above) are stored in a number of zip files in the working

directory. For each language LANG, a zip file called lang-LANG.zip exists and contains an entry for each

word.

The application exposes a command line interface with very simple commands to interact with the

dictionaries.

The section below shows some typical usage.

Running the application with local cache

For your convenience, batch scripts are provided which allow you setup your environment and run the

application. A hint is provided below where double-clicking on the script allows you bypass typing into

the cmd shell.

Every command must be run with the environment properly setup. You will need to do this for every

cmd shell window you open.

set JAVA_HOME=c:\jrmc_3.1

set COHERENCE_HOME=c:\coherence

set CLASSPATH=.;%COHERENCE_HOME%\lib\coherence.jar;%COHERENCE_HOME%\lib\tangosol.jar

set PATH=.;%JAVA_HOME%\bin;%PATH%;

To start, compile the application.

javac -d . app\*.java

Hint: you can just run the app-compile.bat script.

Now generate the dictionaries as zip files containing the words for different languages.This will take

about 5 minutes.

Java -Xms1g Xmx1g app.CreateDictionary

Hint: you can just run the app-create-dict.bat script.

Now run the application using the local cache. Here, the cached data is stored in some Collection objects

on the local java heap.

java -Xms1g -Xmx1500m -Xmanagement app.Main -local

Hint: You can just run the app-local.bat script.

This will open up a prompt as below:

Main>

At the prompt, type help to see the different commands and how to use them

Main> help

Let us get familiar with the application from a user's point of view. At startup, the dictionary has not

been loaded up into memory.

Let us look up information about the word word100003 from the language lang1. Since the dictionary is

not loaded up into memory, the application will retrieve the information from the zip files directly, and

cache just that result into memory.

Main> show lang1 word100003

Now, update the description associated with the word word100003 from the language lang1

Main> update -d lang1 word100003 this new description is cool

Run the show command again to ensure that the cache was updated. Also, check the zip file (lang-

lang1.zip) and look at the entry for word100410, and ensure that the change was written transparently

to the backend zip file. You will notice that updates are slow. Unfortunately, java has poor support for

updating zip files, so we had to completely recreate the zip file for each changed word (this is while

update takes a while).

Load up the whole dictionary for all languages into memory so that we can do some more complex

queries. This may take up to 30 seconds to load all 10 dictionaries.

Main> load

Now, perform a query. Search for words which exist in the language lang1, that are synonymous with

synonym674

Main> find lang1 -s synonym674

Let us do a more complex query. Search for words which exist in the language lang1, that are

synonymous with synonym674 and synonym1093, but opposite (antonym) of antonym948

Main> find lang1 -s synonym674 -s synonym1430 -a antonym2243

For both of these, you got an UnsupportedOperationException because the query support is hard to

implement in our local cache. A database and SQL would have made life much easier but we do not have

access to that for our sample application.

Hint: This is possible and easy using Coherence.

Let us try some more commands.

Look up the languages currently supported in our dictionary

Main> langs

Now look up the stats of the currently loaded cache. This should show you the number of words

currently loaded into the cache for each language.

Main> stats

Explore the application. Understand how it works.

Now, we will go ahead and explore the application to see how it works under the hood.

The application is run from the working directory. For each language LANG, there is a zip file in that

directory called lang-LANG.zip which contains meta-data for each word in the language. Open up lang-

lang1.zip and look at the contents for familiarity.

This working directory is already setup as an Eclipse project. Open up eclipse and import the project into

your workspace. From the file menu, select Import. From the dialog box that comes up, open the

General node in the tree menu and select Existing Project into Workspace. Click Next. Click the

radio button beside Select Root Directory and click the Browse button to select the working

directory. Select the project oow_hol_coherence and click Finish.

Hint: You can just run the eclipse.bat script. The working directory is setup as a pre-configured eclipse

workspace and project.

You should have the project as below:

The java source files all reside under the app directory. Please read through them to get familiar with

what they do.

Record.java This encapsulates a word in a language

Main.java This is the main command line interface that parses the user inputs and calls the

appropriate commands

Worker.java This interface allows us decouple the implementation of the cache from the

application

LocalWorker.java This implementation of Worker uses a local hash map to store the data, and (tries

to) manages the computation itself

Helper.java This contains some shared helper functions

Metrics.java This is a table model containing the languages and memory used for each language

within the cache.

Monitor.java This is a swing UI which regularly gets updated metrics from the cache and displays

as a Swing table.

Once you have a good understanding of the application, feel free to go back to the previous section to

run the lab again and get familiar with the application. The rest of the lab will go a lot smoother once

you get a hang of what the application is trying to accomplish.

Update application to run with distributed cache

Now, the lab gets more interesting. We will walk through a number of steps to update the application so

it runs against the Coherence Data Grid.

In this lab, we aim to achieve the following using Coherence

Store the cached data in multiple external JVMs which look to us as a giant local heap with quick

access

Transparently read the records from the cache, even if they have not been loaded into

Coherence

Transparently let the cache persist updates to the backend data source

Perform queries and computation on the coherence cache

Configure the coherence cluster so that other users on the network do not conflict with us

Efficiently make updates with the minimum amount of network hops

Monitor the application (including the distributed cache) transparently from a single location

To achieve this, we will understand and leverage the following concepts from Coherence

Invocable Maps and Entry Processors

JMX functionality

Read-Through and Write-Through caching strategies

Configuration subsystem

Coherence Services

Partitioned (Distributed) and Near cache topologies

First, define an architecture strategy. In general, Coherence supports using a Partitioned cache where a

number of backups are stored on one or more members in the data grid for failover. In addition, a near

cache can wrap a partitioned cache, so that a copy of the data is stored on the local JVM for each (so

network access may be bypassed). We will leverage both mechanisms for our caching.

The actual words and their metadata are stored in the partitioned (distributed) cache. There will be a

different named cache used for each language. For example, language lang1 will be stored in the cache

called dict-lang1. The list of available languages and some other state we need will be stored in the near

cache (called app-shared). This near cache will actually wrap a different partitioned cache.

We will also use a CacheStore so that read requests to the cache will transparently go to the backend

when a cache miss happens, and updates will also transparently go to the backend zip files when we

want them to.

Configuration Subsystem

Coherence is an extensively configurable system.

At startup, Coherence will find the file tangosol-coherence-override.xml on your classpath. This

configuration is for system-wide settings. In this file, you can configure things like your cluster address,

cluster multicast port, cluster name, etc.

Copy tangosol-coherence-override.xml from the solutions directory into your working directory. Use the

configuration for a local machine restricted cluster, and also define a different distributed cache service

for storing the shared state (e.g. languages).

Once Coherence gets a request to lookup a Cache, it will load up the coherence-cache-config.xml file.

You should configure your caches here, setting up those that will use the distributed (partitioned) cache

separately from those that should use the replicated cache. Copy coherence-cache-config.xml from the

solutions directory into your working directory, and ensure the following:

Two separate cache schemes are defined for dictionary (using distributed scheme) and shared

caches (using near scheme) respectively.

A naming convention is used where caches with names matching dict-* are mapped to the

distributed scheme, while the cache app-shared is mapped to the near scheme.

A cache store is configured which takes the cache name as a parameter. The cache store should

apply to every dict-* cache alone, since these are the only ones which need to load up or persist

data to the backend zip files.

Now that we have gotten the configuration out of the way, let us go about working on the actual

application.

Cache Store

First, create the CacheStore which does the work of reading data during a cache-miss or persisting data

after a cache update. We want an explicit update to the cache to write through to the zip files, but not a

bulk load. This means that every cache put should not write back to the zip files. We can control this by

using a variable stored in the cache itself. The shared cache (app-shared) will be used to store a variable.

During a bulk load, we will set the variable in the cache, and remove it once the bulk load is done. The

cache store will not do a write-through to the backend if the variable is stored. Any other cache puts will

result in a write-through to the backend. Look at the way app\LocalWorker.java reads and writes

individual records to/from the zip files. Implement that same logic in app\DistCacheStore.java.

Hint: Look at the solutions directory for how this is done.

Distributed Cache Worker implementation

Next, create the actual implementation of the Worker interface, similar to the app\LocalWorker.java

implementation, that can interact with Coherence. Call this app\DistWorker.java. In this

implementation, we need to store all variables in the coherence cache, so that anyone in the coherence

cluster can access this. This includes:

languages

dictionaries (words)

Implement the following methods, with guidelines below:

getLanguages store the list of languages in the repl-shared cache and retrieve from there as needed

getRecord Simply retrieve the word from the cache. Coherence will ensure that it checks the

backend zip file if it does not have it, since the cache store has been configured.

update One way to implement this will be to retrieve the Record from Coherence to your local

JVM, make updates on the local JVM, and then send the updated Record over to

Coherence. However, this can cause unnecessary network traffic which could be a

bottleneck if the size of the records is large (especially compared to the change you

want to make). A more efficient way is to send your updates directly to the coherence

node that hosts the data. You achieve this using EntryProcessors and the

InvocableMap.

In addition, this method must be smart enough to write-through to the backend zip

files only when requested. This can be done by setting a variable in the cache before

putting the record in, and the CacheStore will only persist to the zip files if that flag is

set.

bulkInsert For large uploads, it is more efficient to upload to Coherence in batches. Coherence has

a putAll API which can be used for this.

find Unlike the app.LocalWorker (where implementing complex queries without a database

is difficult and un-implemented), Coherence has an extremely powerful query

functionality which can easily simulate complex SQL queries. This is the Coherence

Filters API. An added advantage is that Coherence will run your query in parallel across

all the nodes in the cluster and return your results faster. The more Coherence nodes

you have, the less data each one holds and the less work it does, meaning that your

query performance scales linearly with the number of Coherence nodes.

updateMetrics Coherence has an extensive JMX feature set, where all the management information

can be federated into any number of nodes in the coherence cluster that you deem

should hold federated management information for the cluster. We will leverage this

to keep track of the number of entries in the cache, how these entries are distributed

in the coherence data grid, and how memory is used across the grid.

clear This will remove all entries stored in the cache for a given language

Look in the solutions directory for the full solution for the DistWorker.java implementation.

Finally, update the command line interface app\Main.java to have the -dist command line parameter

switch to using the app.DistWorker implementation. Do this by un-commenting the call that

instantiates DistWorker below.

public static void main(String[] args) throws Exception {

Main m = new Main();

m.worker = new LocalWorker();

for(int i = 0; i < args.length; i++) {

if(args[i].equals("-dist")) {

//m.worker = new DistWorker();

}

}

m.run();

}

Run application with distributed cache

Compile your application as was done in one of the prior sections, by opening up a cmd shell, setting up

your environment and running javac.

Hint: You can run the app-compile.bat script

Now, start up three coherence cache servers, using the command line:

java -Xms256m -Xmx512m -Dtangosol.coherence.management.remote=true

com.tangosol.net.DefaultCacheServer

Hint: You can just run the coherence-cache-server.bat script. Double-click three times to start 3 servers.

This will start up a Coherence cache server configured to expose its management (JMX) information

around the cluster.

Now run the application using the distributed cache implementation, using the command line:

java -Xms256m -Xmx512m -Xmanagement -Dtangosol.coherence.management=all -

Dtangosol.coherence.management.remote=true -Dtangosol.coherence.distributed.localstorage=false

app.Main -dist

Hint: You can just run the app-dist.bat script

This will start the command line interface to your application, setting that JVM as a coherence cluster

member which does not store data, but which collects management information from all the coherence

cluster members and stores it in its JMX MBeanServer. You will thus locally be able to see all the

management information from the whole coherence cluster within your local JVM.

Note that once the data has been loaded into Coherence, you can run multiple clients and have them all

share the distributed in-memory data grid.

At the prompt, run stats which will pop up a Swing UI which updates itself as the Coherence cluster

membership and contents change. From this Swing UI table, you can see the membership of the

coherence cluster and how much of the data each member holds in near-real time (the Swing UI

updates itself every 5 seconds).

Main> stats

As mentioned above, this swing table updates itself every five seconds, with the JMX information which

has been federated at your local JVM. Each row in it represents how the amount of data stored by each

coherence server on behalf of the cluster, and the memory being used in MB. For example, in the

screenshot above, the coherence JVM with node-id 1 is the primary store for 10089 words of lang2,

10075 words for lang1, 9924 words for lang3, and uses 114MB of memory. The next rows show how

much coherence JVMs with node id 2 and 4 hold.

As you add or shutdown coherence cache server, and even as you bulk-load the records into coherence,

monitor this Swing UI and see how Coherence automatically distributes the cached data. Look at

app\DistMonitor.java for the full implementation.

Also, you can open up JConsole or JRMC to look at the rich set of JMX management metrics exposed by

Coherence.

Hint: You can run the jrockit-mc.bat script

Once that is done, run the other commands as was shown in the previous section. A sampling of those

commands is below.

Main> help

Main> langs

Main> show lang1 word100003

Main> update -d lang1 word100003 this new description is cool

Main> load

Main> find lang1 -s synonym674

Main> find lang1 -s synonym674 -s synonym1430 -a antonym2243

Main> exit

To show the beauty of this distributed solution, run another instance of the client. At the prompt, just

run the find command. You will see that it works. Any new client does not have to load up the data,

since all the data and state is stored externally in the distributed cache.

Hint: You can run the app-dist.bat script

Test failover and redundancy

You can test failover and redundancy of your Coherence implementation by shutting down some

instances and bringing others back up. You can do this by typing Ctrl-C in some of the Coherence Cache

Server windows, and/or bringing some new Coherence servers back up.

Hint: You can use the coherence-cache-server.bat script.

Watch the windows of the coherence cache servers that are live. Notice how the coherence cluster

automatically rebalances the data. Watch the Swing UI to see the updated stats on how the data is

rebalanced and memory used in the cache servers.

Shutdown everything

To shutdown everything, type Ctrl-C in all your open windows.