56

Introducing Apache Geode and Spring Data GemFire

Embed Size (px)

Citation preview

Page 1: Introducing Apache Geode and Spring Data GemFire
Page 2: Introducing Apache Geode and Spring Data GemFire

2© 2014 Pivotal Software, Inc. All rights reserved. 2© 2014 Pivotal Software, Inc. All rights reserved.

Introducing Apache Geode with Spring Data GemFireJohn Blum / Luke Shannon

24 June 2015

Page 3: Introducing Apache Geode and Spring Data GemFire

3© 2014 Pivotal Software, Inc. All rights reserved.

Agenda

What is Apache Geode? Apache Geode Technical Specification High-level Implementation Details Spring Data GemFire Q&A

Page 4: Introducing Apache Geode and Spring Data GemFire

4© 2014 Pivotal Software, Inc. All rights reserved.

Pivotal GemFireWhat is Apache Geode?

In a nutshell…– Open Source core of Pivotal GemFire– Apache Incubator

▪ https://wiki.apache.org/incubator/GeodeProposal

Page 5: Introducing Apache Geode and Spring Data GemFire

5© 2014 Pivotal Software, Inc. All rights reserved.

What is Apache Geode?One definition is…A distributed, in-memory compute and data management platform that elastically scales to achieve high-throughput, low-latency access to big, fast data in order to power business critical, analytical applications in real-time.

Nodes

Ops / SecLinear scalability

Elastic capacity +/-

Latency optimizeddata distribution

Page 6: Introducing Apache Geode and Spring Data GemFire

6© 2014 Pivotal Software, Inc. All rights reserved.

What is Apache Geode? Technically…

– Distributed, in-memory database (IMDG) with ACID properties

– Transactional Key/Value Store (Cache)

– Glorified-version of ConcurrentHashMap

– Message Bus with guaranteed message delivery▪ Reliable, asynchronous event notifications

Page 7: Introducing Apache Geode and Spring Data GemFire

7© 2014 Pivotal Software, Inc. All rights reserved.

What Apache Geode Does? Enables new or existing Java applications to operate at cloud-scale

with predictable, low-latencies and be highly-available without sacrificing consistency.

Applications can transact and analyze Big Data in real-time to garner insights and achieve meaningful and impactful business results.

App

App

App

App

Page 8: Introducing Apache Geode and Spring Data GemFire

8© 2014 Pivotal Software, Inc. All rights reserved.

Apache Geode Characteristics In-Memory

Distributed

Consistent

Highly-Available / Fault Tolerant

High-Throughput / Low, Predictable Latency

High Scalability

Data-aware Parallel Compute

Page 9: Introducing Apache Geode and Spring Data GemFire

9© 2014 Pivotal Software, Inc. All rights reserved.

How are Apache Geode’s features implemented?

Page 10: Introducing Apache Geode and Spring Data GemFire

10© 2014 Pivotal Software, Inc. All rights reserved.

Memory Geode stores data in memory…

– JVM Heap– Off-Heap (Native Memory)

Geode (optionally) stores data to disk– Persistence + Overflow

Page 11: Introducing Apache Geode and Spring Data GemFire

11© 2014 Pivotal Software, Inc. All rights reserved.

ManagementMemory

Eviction (LRU)

Expiration (Region & Entry TTL/TTI)

Automatic resource management (critical & eviction Heap %)

Region Compression (Snappy)

JVM GC Tuning

Page 12: Introducing Apache Geode and Spring Data GemFire

12© 2014 Pivotal Software, Inc. All rights reserved.

IMDG An In-Memory Data Grid (IMDG), such as Apache Geode…

– Pools system resources across multiple nodes in a cluster to manage both application state and behavior.

– Memory, CPU, Network and (optional) Disk

Page 13: Introducing Apache Geode and Spring Data GemFire

13© 2014 Pivotal Software, Inc. All rights reserved.

High-Throughput / Low-Latency Geode uses concurrent, in-memory data structures with a highly

optimized distribution infrastructure to minimize context switching and contention.– synchronous/asynchronous replication for high read throughput

– partitioning for both high read and write throughput

Page 14: Introducing Apache Geode and Spring Data GemFire

14© 2014 Pivotal Software, Inc. All rights reserved.

Shared-Nothing Disk Persistence Architecture Manages data in local disk files on each node in the cluster

independently from other nodes– Members are self-sufficient; Members do not share memory or disk;– Data/Transactions appended to Oplog files– Compaction

Design goals (all about In-Memory)…– Minimize IO operations (Write-Behind vs. Write-Through)– Minimize locking– Minimize serialization

Page 15: Introducing Apache Geode and Spring Data GemFire

15© 2014 Pivotal Software, Inc. All rights reserved.

REPLICATE & PARTITION RegionsHigh Availability

Data is partitioned across nodes in the cluster and distributed uniformly– Rebalanced as nodes join/leave– Redundant copies

Page 16: Introducing Apache Geode and Spring Data GemFire

16© 2014 Pivotal Software, Inc. All rights reserved.

Persistence – Shared NothingServer 3Server 2

Bucket 3*primary*

Bucket 1secondary

Bucket 3secondary

Bucket 2*primary*

Server 1

Bucket 2secondary

Bucket 1*primary*

Page 17: Introducing Apache Geode and Spring Data GemFire

17© 2014 Pivotal Software, Inc. All rights reserved.

Persistence – Shared NothingServer 1 Server 3Server 2

Bucket 2secondary

Bucket 1*primary*

Bucket 3*primary*

Bucket 1secondary

Bucket 3secondary

Bucket 2*primary*

What if a server crashes?

Page 18: Introducing Apache Geode and Spring Data GemFire

18© 2014 Pivotal Software, Inc. All rights reserved.

Persistence – Shared NothingServer 1 Server 3Server 2

Bucket 2secondary

Bucket 1*primary*

Bucket 3*primary*

Bucket 1*primary*

Bucket 3secondary

Bucket 2*primary*

Bucket 2secondary

Primary Failover

RestoreRedundancy

Bucket 1secondary

Page 19: Introducing Apache Geode and Spring Data GemFire

19© 2014 Pivotal Software, Inc. All rights reserved.

Network Partitions

Page 20: Introducing Apache Geode and Spring Data GemFire

20© 2014 Pivotal Software, Inc. All rights reserved.

Partition RegionsConfigurable Consistency

Redundant Copies – 0 to 3

Colocation

Total Number of Buckets– 113 (Default)

Local Max Memory– 90% of Heap

Total Max Memory– All Members, Both Primary/Secondary

Copies

Page 21: Introducing Apache Geode and Spring Data GemFire

21© 2014 Pivotal Software, Inc. All rights reserved.

REPLICATE vs. PARTITIONConfigurable Consistency

Partition Regions– Updates routed through primary bucket– Holds lock on key when distributing updates

Replicate Regions– Possible conflicts due to out-of-order updates– Conflict checking (--concurrency-checks-enabled)

ReplicatedPartitioned Disabled

Page 22: Introducing Apache Geode and Spring Data GemFire

22© 2014 Pivotal Software, Inc. All rights reserved.

Apache Geode Topologies

Page 23: Introducing Apache Geode and Spring Data GemFire

23© 2014 Pivotal Software, Inc. All rights reserved.

Peer-to-Peer (P2P)Apache Geode Topologies

Configuration with the Cache embedded in the application process– Close proximity of application logic and data

Page 24: Introducing Apache Geode and Spring Data GemFire

24© 2014 Pivotal Software, Inc. All rights reserved.

Client/ServerApache Geode Topologies

Single-Hop Data Access

Continuously load balanced between servers to get the best response-times

`

Automatic fail-over when servers crash or become unresponsive

Page 25: Introducing Apache Geode and Spring Data GemFire

25© 2014 Pivotal Software, Inc. All rights reserved.

WANApache Geode Topologies

Site-to-Site Replication for DR

Page 26: Introducing Apache Geode and Spring Data GemFire

26© 2014 Pivotal Software, Inc. All rights reserved.

Use CasesApache Geode

Persistent OLTP/OLAP– System of Record (SOR)

Caching with JSR-107 (JCache) or Spring Cache Abstraction HTTP Session State Management

– Caching and Replication

Distributed L2 Caching for Hibernate Memcached Server (Gemcached)

Page 27: Introducing Apache Geode and Spring Data GemFire

27© 2014 Pivotal Software, Inc. All rights reserved.

Additional FeaturesApache Geode

Functions– Application logic is distributed, routed and run in parallel with close proximity to the data required by the

application.

Serialization (PDX)

Delta-propagation – difference of new/old value

JTA-compliant Transactions

Security – SSL, Auth

Page 28: Introducing Apache Geode and Spring Data GemFire

28© 2014 Pivotal Software, Inc. All rights reserved.

RoadmapApache Geode

HDFS persistence Off-Heap storage Lucene search/indexing Spark integration Cloud Foundry service

Page 29: Introducing Apache Geode and Spring Data GemFire

29© 2014 Pivotal Software, Inc. All rights reserved.

InterfaceApache Geode

Language Support– Native: Java, C/C++, C#– Non-native: JavaScript, Python,

Ruby, Scala, etc

API– Java HashMap (Region)– OQL / Function– Memcache client– REST API– Spring Data GemFire

Data Structure– User-defined Classes / POJOs– Portable Data Exchange (PDX) – Documents (JSON)

Powerful Data Access– Object Query Language (OQL)– Data-aware, Scatter-Gather

Functions– Publish/Scribe framework for RI/CQs

with reliable, async event queues

Page 30: Introducing Apache Geode and Spring Data GemFire

30© 2014 Pivotal Software, Inc. All rights reserved. 30© 2014 Pivotal Software, Inc. All rights reserved.

Spring Data GemFirehttp://projects.spring.io/spring-data-gemfire/

Page 31: Introducing Apache Geode and Spring Data GemFire

31© 2014 Pivotal Software, Inc. All rights reserved.

So… Why Spring?

Page 32: Introducing Apache Geode and Spring Data GemFire

32© 2014 Pivotal Software, Inc. All rights reserved.

“Simple things should be simple; complex things should be possible”

– Alan Kay

Page 33: Introducing Apache Geode and Spring Data GemFire

33© 2014 Pivotal Software, Inc. All rights reserved.

Spring Data GemFire (SDG)Consistently applies Spring's powerful, non-invasive programming model to simplify configuration and development with Apache Geode.

Spring Ecosystem Integration…– Spring Cache Abstraction / Transaction Management– Spring Data Commons + REST– Spring Integration (Inbound/Outbound Channel Adapters)– Spring XD (Input Source and Sinks)

Page 34: Introducing Apache Geode and Spring Data GemFire

34© 2014 Pivotal Software, Inc. All rights reserved.

Apache Geode combined with Spring Data GemFire can be used as JSR-107 (JCache) caching provider

+ +

Page 35: Introducing Apache Geode and Spring Data GemFire

35© 2014 Pivotal Software, Inc. All rights reserved.

GRAILS

Full-stack, Web

XD

Stream, Taps, Jobs

BOOT

Bootable, Minimal, Ops-Ready

Big, Fast,

FlexibleData Web,

Integration,Batch

WEB

Controllers, REST,WebSocket

INTEGRATION

Channels, Adapters,Filters, Transformers

BATCH

Jobs, Steps,Readers, Writers

BIG DATA

Ingestion, Export,Orchestration, Hadoop

DATA

NON-RELATIONALRELATIONAL

CORE

GROOVYFRAMEWORK SECURITY REACTOR

Page 36: Introducing Apache Geode and Spring Data GemFire

36© 2014 Pivotal Software, Inc. All rights reserved.

Use Cases / ConfigurationsSpring Data GemFire

Apache Geode Configuration / Bootstrapping– Replacement for cache.xml; Can be used with Cluster Configuration

Application Peer (Cache) Application Client (ClientCache)

Page 37: Introducing Apache Geode and Spring Data GemFire

37© 2014 Pivotal Software, Inc. All rights reserved.

Examples…SDG-based Configuration

Page 38: Introducing Apache Geode and Spring Data GemFire

38© 2014 Pivotal Software, Inc. All rights reserved.

with SpringBootstrapping Apache Geode

@SpringBootApplication@ImportResource("/spring-data-gemfire-cache.xml")@EnableGemfireRepositories@EnableTransactionManagementpublic class SampleDataGemFireApplication { public static void main(String[] args) { SpringApplication.run(SampleDataGemFireApplication.class, args); }}

Or…gfsh>start server –name=Example … --spring-xml-location=“<classpath-to-spring-application-context.xml>”

Page 39: Introducing Apache Geode and Spring Data GemFire

39© 2014 Pivotal Software, Inc. All rights reserved.

Basic (Region) Data AccessData Access with Spring

@Repositorypublic class CustomerDao {

@Resource(name = “Example”) private Region<?, ?> example; …}

Example…

Page 40: Introducing Apache Geode and Spring Data GemFire

40© 2014 Pivotal Software, Inc. All rights reserved.

GemfireTemplateData Access with Spring Data GemFire

Advantages…– Simple, Convenient Data Access (CRUD, OQL, Function)– Protects developer from GemFire/Geode API changes – Exception Translation into Spring DAO Exception Hierarchy– Transaction Management

<bean name=“customersTemplate” class=“org.springframework.data.gemfire.GemfireTemplate” p:region-ref=“Customers”>

Page 41: Introducing Apache Geode and Spring Data GemFire

41© 2014 Pivotal Software, Inc. All rights reserved.

GemfireRepositoryData Access with Repositories

public interface CustomerRepository extends GemfireRepository<Customer, Long> {

}

// Spring context.xml<gfe-data:repositories base-package="org.example.app.customer.dao.repo"> <repo:include-filter type="assignable" expression=”org.example.app.customer.dao.repo.CustomerRepository"/></gfe-data:repositories>

Example…

Page 42: Introducing Apache Geode and Spring Data GemFire

42© 2014 Pivotal Software, Inc. All rights reserved.

Implementation

Annotation-based Function Implementation/Execution

package org.example.app.functions;

import …

@Componentpublic class ExampleFunctions {

@GemfireFunction public Customer update(…) { … } …}

Example…

(Optional)…

<bean class=“org.example….ExampleFunctions”/>

Page 43: Introducing Apache Geode and Spring Data GemFire

43© 2014 Pivotal Software, Inc. All rights reserved.

Execution

Annotation-based Function Implementation/Execution

package org.example.app.function.executions;

import …

@OnRegion(“Example”)public interface ExampleFunctionExecution { Customer update(…);}

And…<gfe:annotation-driven/>

<gfe-data:function-executions base-package="org.example.app.function.executions">

Page 44: Introducing Apache Geode and Spring Data GemFire

44© 2014 Pivotal Software, Inc. All rights reserved.

Execution

Annotation-based Function Implementation/Execution

Additionally…– @OnServer(s), @OnMember(s)

Examples…

Page 45: Introducing Apache Geode and Spring Data GemFire

45© 2014 Pivotal Software, Inc. All rights reserved.

JSR-107 (JCache), Spring Cache AbstractionGeode Caching Provider

With @Cacheable And @CachePut, @CacheEvict(allEntries=[true|false]), @CacheConfig

– http://docs.spring.io/spring/docs/current/spring-framework-reference/htmlsingle/#cache-jsr-107

Or with JSR-107 using…– @CacheResult, @CachePut, @CacheRemove, @CacheRemoveAll,

@CacheDefaults

Page 46: Introducing Apache Geode and Spring Data GemFire

46© 2014 Pivotal Software, Inc. All rights reserved. 46© 2014 Pivotal Software, Inc. All rights reserved.

DemoBy Luke Shannon

Page 47: Introducing Apache Geode and Spring Data GemFire

47© 2014 Pivotal Software, Inc. All rights reserved.

On the horizon…

Page 48: Introducing Apache Geode and Spring Data GemFire

48© 2014 Pivotal Software, Inc. All rights reserved.

RoadmapSpring Data GemFire

Support for GemFire 8.2 & 9.0– HDFS, Off-Heap, Apache Lucene, Spark

Support for pre-defined data sets using Snapshot Service– SGG-408

Expiration annotation support with CustomExpiry– SGF-404

Spring Session Adapter for GemFire/Geode– SGF-373

Page 49: Introducing Apache Geode and Spring Data GemFire

49© 2014 Pivotal Software, Inc. All rights reserved.

Roadmap Continued…Spring Data GemFire

Repository Query method annotation extensions...– SGF-392

public interface ExampleRepository implements GemfireRepository<Customer, Long> {

@Trace @Limit(20) @Import(“org.example.app.domain.SomeOtherType”) List<Customer> findByLastname(String lastName);

}

Page 50: Introducing Apache Geode and Spring Data GemFire

50© 2014 Pivotal Software, Inc. All rights reserved.

Get Involved…

Page 51: Introducing Apache Geode and Spring Data GemFire

51© 2014 Pivotal Software, Inc. All rights reserved.

Apache GeodeContribute

In addition to developing Spring applications with Apache Geode, now is a perfect time to get involved in the Apache Geode community and contribute growing Apache project from the ground-up.– Ways to participate…

http://geode.incubator.apache.org/contribute/

Page 52: Introducing Apache Geode and Spring Data GemFire

52© 2014 Pivotal Software, Inc. All rights reserved.

Finally…

Page 53: Introducing Apache Geode and Spring Data GemFire

53© 2014 Pivotal Software, Inc. All rights reserved.

Washington, DC – September 14 – 17, 2015SpringOne2GX

Early Bird price registration ends August 14th, Register Now!– https://2015.event.springone2gx.com/register

Building High-Scalable Spring Applications with Distributed In-Memory Data Grids.– Luke Shannon & John Blum– Tuesday, September 15th, 2015– https://2015.event.springone2gx.com/schedule/sessions/

building_highly_scalable_spring_applications_with_in_memory_distributed_data_grids.html

Page 54: Introducing Apache Geode and Spring Data GemFire

54© 2014 Pivotal Software, Inc. All rights reserved.

References http://geode.incubator.apache.org/ http://geode.incubator.apache.org/contribute/ https://github.com/project-geode/docs/wiki https://github.com/apache/incubator-geode https://pivotal.io/big-data/pivotal-gemfire http://gemfire.docs.pivotal.io/latest/userguide/index.html#getting_started/topics/gemfire_overview.html http://gemfire.docs.pivotal.io/latest/javadocs/japi/index.html http://stackoverflow.com/questions/tagged/geode+or+gemfire http://projects.spring.io/spring-data-gemfire/ http://docs.spring.io/spring-data-gemfire/docs/1.6.0.RELEASE/reference/html/ http://docs.spring.io/spring-data-gemfire/docs/current/reference/html/ http://docs.spring.io/spring-data-gemfire/docs/current/api/ https://github.com/spring-projects/spring-data-gemfire https://github.com/spring-projects/spring-gemfire-examples http://stackoverflow.com/questions/tagged/spring-data-gemfire

Page 55: Introducing Apache Geode and Spring Data GemFire

55© 2014 Pivotal Software, Inc. All rights reserved. 55© 2014 Pivotal Software, Inc. All rights reserved.

Q&AAny questions…

Page 56: Introducing Apache Geode and Spring Data GemFire

56© 2014 Pivotal Software, Inc. All rights reserved. 56© 2014 Pivotal Software, Inc. All rights reserved.

Thank you!