GAE Developer - Day3Simon @ MiCloud2014Q1

Last Class

● Datastore basic● Datastore operation● Restrictions● Transaction

Create an Entity with KeyEntity job = new Entity("User", "simonsu@mitac.com.tw");

job.setProperty("name", "Simon Su");

job.setProperty("start", "20140103");

job.setProperty("create", new Date());

try {

Key k = ds.put(job);

out.println("Done..." + k.getId());

} catch (Exception e) {

e.printStackTrace();

}

Create an Entity with AncestorKey userKey = KeyFactory.createKey("User", "simonsu@mitac.com.tw");

Entity job = new Entity("Jobs", userKey);

job.setProperty("name", "engineer");

job.setProperty("start", new Date());

try {

Key k = ds.put(job);

out.println("Done..." + k.getId());

} catch (Exception e) {

e.printStackTrace();

}

Get data with KeyDatastoreService ds = DatastoreServiceFactory.getDatastoreService();

Key key = KeyFactory.createKey("User", "simonsu@mitac.com.tw");

try {

Entity ent = ds.get(key);

out.print(ent);

Map m = ent.getProperties();

out.println(m.toString());

} catch (EntityNotFoundException e) {

e.printStackTrace();

}

Get data with AncestorQuery q = new Query("Jobs");

Key ancestor = KeyFactory.createKey("User", "simonsu@mitac.com.tw");

q.setAncestor(ancestor);

PreparedQuery results = ds.prepare(q);

Iterator iter = results.asIterable().iterator();

while(iter.hasNext()) {

Entity ent = (Entity) iter.next();

out.println(ent);

}

Today

● Memcache basic● Task Queue concept● Push Queue● Pull Queue

Memcache

Why memcache

● Improve Application Performance● Reduce Application Cost● Caching for Read heavy operations● Caching In Front of Datastore● Semi-durable Shared state Across App

Instances

Memcache - operationMemcacheService cache = MemcacheServiceFactory.getMemcacheService();

cache.setErrorHandler(

ErrorHandlers.getConsistentLogAndContinue(Level.INFO));

// read from cache

value = (byte[]) cache.get(key);

// save to cache

if (value == null) {

// ........

cache.put(key, value);

}

Async memcache - get// Using the asynchronous cache

AsyncMemcacheService asyncCache =

MemcacheServiceFactory.getAsyncMemcacheService();

asyncCache.setErrorHandler(

ErrorHandlers.getConsistentLogAndContinue(Level.INFO));

// read from cache

Future<Object> futureValue = asyncCache.get(key);

// ... do other work in parallel to cache retrieval

value = (byte[]) futureValue.get();

Async memcache - putif (value == null) {

// get value from other source

// ........

// asynchronously populate the cache

// Returns a Future<Void> which can be used to block until completion

asyncCache.put(key, value);

}

Using jcache - initimport java.util.Collections;

import net.sf.jsr107cache.*;

Cache cache;

try {

CacheFactory cacheFactory =

CacheManager.getInstance().getCacheFactory();

cache = cacheFactory.createCache(Collections.emptyMap());

} catch (CacheException e) {

// ...

}

Using jcache - get & putString key; // ...

byte[] value; // ...

// Put the value into the cache.

cache.put(key, value);

// Get the value from the cache.

value = (byte[]) cache.get(key);

Testing with ab benchmark

ab -c 50 -n 50 http://[app-address]/[servlet_path]

or

ab -c 50 -t 50 http://[app-address]/[servlet_path]

Other Operation● getAll(), putAll(), deleteAll()

A single read or write operation for multiple memcache entries

● increment(key, delta), incrementAll(...)Provide atomic increment of numeric value(s)

● getIdentifiable(), putIfUntouched()A mechanism to update a value consistently by concurrent requests

Memcache Caveat

● Memcache Is Volatile● Performance suggest: Batch size < 32 MB● Dedicated memcache (pay): 10K OPS/s/GB,

range 1-20 GB

Task Queue

Why Task Queue● Frontend restriction of 60 second limit● Need longer-running tasks● Asynchronously outside front end request● Best practice:

○ Email○ Write datastore○ Web crawler○ Backend jobs

Task Queue Types● By name

○ Default Queue (Push Queue)○ Named Queue

● By type○ Push queues

■ tasks execute automatically■ only for App Engine app

○ Pull queues■ tasks wait to be leased■ available to workers outside the app■ tasks can be batched

queue.xml<queue-entries>

<queue>

<name>Queue-Name</name>

<rate>1/s</rate>

...

</queue>

<queue>

<name>Queue-Name2</name>

<mode>pull</mode>

...

</queue>

</queue-entries>

Push Queue

Pull Queue

Task Queue Manage Console

Task Queue - Default Push Queue

Queue queue = QueueFactory.getDefaultQueue(); //named queue: //QueueFactory.getQueue("Qname");

TaskOptions task = TaskOptions.Builder .withUrl("/path-to-my-worker") .param(key, value);

queue.add(task);

Reachable Servlet

Detail Configure

● Task queues use token bucket algorithm [ref]

● rate - usual rate● bucket-size - cap on peak demand● max concurrent requests● Retry options

○ task-retry-limit - min retries○ task-age-limit - min elapsed time to keep retrying

(Both task-retry-limit and task-age-limit are reached, task is deleted)○ min-backoff-seconds -- min delay between retries○ max-backoff-seconds -- max delay between retries○ max-doublings -- max times to double delay

● more parameters see DOC

queue.xml parameters

Task Queue - With TransactionDatastoreService ds = DatastoreServiceFactory.getDatastoreService();Queue queue = QueueFactory.getDefaultQueue();try { Transaction txn = ds.beginTransaction();

// … other operations queue.add(TaskOptions.Builder.withUrl("/path-to-my-worker")); // … other operations

txn.commit();} catch (DatastoreFailureException e) { // … exception handle}

Pull Queue - Create Queue

Queue queue =

QueueFactory.getQueue("queue_name");

TaskOptions task =

TaskOptions.Builder

.withMethod(TaskOptions.Method.PULL)

.payload(payload);

queue.add(task);

Pull Queue - Lease

Queue queue =

QueueFactory.getQueue("queue_name");

tasks = q.leaseTasks(

duration,

TimeUnit.SECONDS,

how-many);

Pull Queue - Delete Task

//delete named taskqueue.deleteTask("task_ name");

//purge all tasks from queueQueueFactory.getQueue("queue_name").purge();

Task Queue ACL - queue.xml

<queue>

<name>pull-queue1</name>

<mode>pull</mode>

<acl>

<user-email>me@gmail.com</user-email>

<writer-email>you@gmail.com</writer-email>

<writer-email>she@gmail.com</writer-email>

</acl>

</queue>

Cron

Cron

<cronentries>

<cron>

<url>/recache</url>

<description>

Put your description here

</description>

<schedule>every 5 minutes</schedule>

</cron>

</cronentries>

● every 12 hours● every 5 minutes from 10:00 to 14:00● 2nd,third mon,wed,thu of march 17:00● every monday 09:00● 1st monday of sep,oct,nov 17:00● every day 00:00● every N (hours|mins|minutes) ["from" (time) "to" (time)]● every 2 hours synchronized

Cron Schedule Format

Cron Manage Console

Restrictions

● 20 cron jobs for free apps● 100 for paid apps

Task Queue / Cron Reference● Task Queues Developer Guide (Java) (Python)

● Task Queue Configuration (Java) (Python)

● Task Queue API Ref (JavaDoc) (Python)

● Task Queue REST API Ref

● Scheduled Tasks with Cron (Java) (Python)

● Task queue quota and limits

● Get started with Task API article (Python)

● Google I/O 2011: Putting Task Queues to Work (video)