Introduction to Datastore

Assoc.Prof. Dr.Thanachart NumnondaAsst.Prof. Thanisa Kruawaisayawan

www.imcinstitute.comJuly 2012

Agenda

What is DataStore?Using DataStoreJPA in DataStore

What is DataStore?

What is Datastore?Google App Engine Datastore is a schema-less persistence

system, whose fundamental persistence unit is called Entity, composed by an immutable Key and a collection of mutable properties.

Entities can be created, updated, deleted, loaded by key and queried for properties values.

DataStore is consistent and transactional, with support to current transaction.

The DataStoreThe Datastore is not a relational database nor a

façade.Relational database technology doesn’t scale

horizontally– Connection pools, shared caching are a problem

The Datastore is one of many public APIs used for accessing Google’s

The DataStore

The DataStore

The DataStore : OperationsTransactions and Index are based on MegaTable.File persistence it's done with Google File System

(GFS).It's distributed by Chubby, a lock service for loosely-

coupled distributed systems.

BigTableBigTable is a compressed, high performance, and

proprietary database system built on Google File System (GFS), Chubby Lock Service, and a few other Google programs

Currently not distributed or used outside of Google.BigTable development began in 2004. and is now used

by a number of Google application Google Earth, Google Map, Gmail, Youtube, etc..

BigTable : DesignBigTable is a fast and extremely large-scale DBMS. It is a sparse, distributed multi-dimensional sorted map,

sharing characteristics of both row-oriented and column-oriented databases.sparse because only "not null" values are persisteddistributed in Google cloud persistent on Google File Systemmultidimensional in columns valuesordered lexicographically by key

BigTable : Design

Tables are optimized for GFS by being split into multiple tablets - segments of the table.

BigTable is designed to scale into the petabyte.Each table has multiple dimensions (one of which is a

feld for time, allowing for versioning and garbage collection).

It allows an infnite number of rows and columns.

Google File SystemGFS is a proprietary distributed fle system developed

by Google.It is designed to provide effcient, reliable access to

data using large clusters of commodity hardware.GFS grew out of an earlier Google effort, BigFiles,

developed by Larry Page and Sergey Brin in the early days of Google, while it was still located in Stanford.

Using DataStore

DataStore OperationsDatastore operations are defned around entities (data

models) which are objects with one or more propertiesTypes: string, user, Boolean, and so onEntities may be recursive or self-referential

Entity relationships are one-to-many or many-to-many.Entities may be fxed or grow as needed.

DataStore Storage ModelEvery entity is of a particular kindEntities in a kind need not have the same propertiesOne entity may have different “columns” from another in

the same kind!Unique IDs are automatically assigned unless the user

defnes a key_name

Compare DataStore with Others

DataStore Storage ModelBasic unit of storage is an Entity consisting of

Kind (table)Key (primary key)Entity Group (partition)0..N typed Properties (columns)

Datastore QuotasEach call to Datastore counts towards the quotaThe amount of data cannot exceed the billable

Includes properties and keys but not the indices

CPU and Datastore CPU time quotas apply

Using the DatastoreApplications may access the Datastore using the JDO

or the JPA classes.The JDO and JPA classes are abstracted using the

DataNucleus APIOpen source Not very popular Support for Java standards Poor documentation

JPA in DataStore

Setting Up JPAThe JPA and datastore JARs must be in the app's

war/WEB-INF/lib/ directory. A confguration fle named persistence.xml must be in

the app's war/WEB-INF/classes/META-INF/ directory,A confguration fle tells JPA to use the App Engine

datastore.The appengine-api.jar must also be in the war/WEB-

INF/lib/ directory.

persistence.xml: Example

<?xml version="1.0" encoding="UTF-8"?>

<persistence version="1.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">

<persistence-unit name="thaijavaappPU" transaction-type="RESOURCE_LOCAL">

<provider>org.datanucleus.store.appengine.jpa.DatastorePersistenceProvider</provider>

<non-jta-data-source/>

<properties>

<property name="datanucleus.ConnectionURL" value="appengine"/>

<property name="datanucleus.NontransactionalRead" value="true"/>

<property name="datanucleus.NontransactionalWrite" value="true"/>

</properties>

</persistence-unit>

</persistence>

<?xml version="1.0" encoding="UTF-8"?>

<persistence version="1.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">

<persistence-unit name="thaijavaappPU" transaction-type="RESOURCE_LOCAL">

<provider>org.datanucleus.store.appengine.jpa.DatastorePersistenceProvider</provider>

<non-jta-data-source/>

<properties>

<property name="datanucleus.ConnectionURL" value="appengine"/>

<property name="datanucleus.NontransactionalRead" value="true"/>

<property name="datanucleus.NontransactionalWrite" value="true"/>

</properties>

</persistence-unit>

</persistence>

Getting an EntityManager InstanceAn app interacts with JPA using an instance of the EntityManager.

import javax.persistence.EntityManagerFactory;

import javax.persistence.Persistence;

public class EMF {

private static final EntityManagerFactory emfInstance =

Persistence.createEntityManagerFactory("transactions-optional");

public static EntityManagerFactory get() {

return emfInstance;

}

}

import javax.persistence.EntityManagerFactory;

import javax.persistence.Persistence;

public class EMF {

private static final EntityManagerFactory emfInstance =

Persistence.createEntityManagerFactory("transactions-optional");

public static EntityManagerFactory get() {

return emfInstance;

}

}

Entity Class : Example@Entity

public class GuestList implements Serializable {

…

@Id

private String id;

@Basic

private User author;

private String content;

@Temporal(javax.persistence.TemporalType.DATE)

private Date visitDate;

…

// Getter and Setter methods

}

@Entity

public class GuestList implements Serializable {

…

@Id

private String id;

@Basic

private User author;

private String content;

@Temporal(javax.persistence.TemporalType.DATE)

private Date visitDate;

…

// Getter and Setter methods

}

Queries and IndicesA query operates on every entity of a given kind.

Specify zero or more sort ordersSpecify zero or more flters on property values

Indices are defned in the App Engine confguration flesResults are fetched directly from these indices; no indices are

created on the flyWEB-INF/datastore-indexes.xml - non-standard fles

Normalization is not recommendedOptimization techniques for RDBMSs may result in poor

Datastore performance!

Query : Example EntityManager em = EMF.get().createEntityManager();

try {

Query query = em.createQuery("SELECT o FROM GuestList AS o");

@SuppressWarnings("unchecked")

List<GuestList> results = (List<GuestList>) query.getResultList();

for (Object obj : results) {

GuestList guest = (GuestList) obj;

String nickname = guest.getAuthor().getNickname();

out.println(nickname + " " + guest.getId());

}

} catch(Exception ex) {

out.println(ex);

}

EntityManager em = EMF.get().createEntityManager();

try {

Query query = em.createQuery("SELECT o FROM GuestList AS o");

@SuppressWarnings("unchecked")

List<GuestList> results = (List<GuestList>) query.getResultList();

for (Object obj : results) {

GuestList guest = (GuestList) obj;

String nickname = guest.getAuthor().getNickname();

out.println(nickname + " " + guest.getId());

}

} catch(Exception ex) {

out.println(ex);

}

Entity Relationships

Models association between entities.There are four types of relationship multiplicities:

@OneToOne@OneToMany@ManyToOne

Supports unidirectional as well as bidirectional relationshipsUnidirectional relationship: Entity A references B, but B doesn't

reference A.

Example : ManyToOne Mapping

Example : OneToMany Mapping

Transactions and Entity GroupsTransaction = Group of Datastore operations that either

succeed or failEntity groups are required because all grouped entities are

stored in the same Datastore nodeAn entity may be either created or modifed once per

transactionTransactions may fail if a different user or process tries an

update in the same group at the same timeUsers decide whether to retry or roll the transaction back

Transaction in JPA : Example

Book book = em.find(Book.class, "9780596156732");

BookReview bookReview = new BookReview();

bookReview.rating = 5;

book.getBookReviews().add(bookReview);

Transaction txn = em.getTransaction();

txn.begin();

try {

book = em.merge(book);

txn.commit();

} finally {

if (txn.isActive()) {

txn.rollback();

}

}

Book book = em.find(Book.class, "9780596156732");

BookReview bookReview = new BookReview();

bookReview.rating = 5;

book.getBookReviews().add(bookReview);

Transaction txn = em.getTransaction();

txn.begin();

try {

book = em.merge(book);

txn.commit();

} finally {

if (txn.isActive()) {

txn.rollback();

}

}

Unsupported Features of JPAOwned many-to-many relationships, and unowned

relationships."Join" queries.Aggregation queries (group by, having, sum, avg, max, min) Polymorphic queries.

ResourcesGoogle App Engine for Java HOWTO, Andrew Lombardi, Mar

2010The Softer Side Of Schemas, Max Ross, May 2009Official Google App Engine Tutorial,

http://code.google.com/appengine/docs/java/gettingstarted/Programming Google App Engine, Don Sanderson, O'Reilly,

2010

Thank you

thananum@gmail.comwww.facebook.com/imcinstitute

www.imcinstitute.com