using Queue Servicefor batch processing

Job Queue

data source

convert the data into job and sent to queue

1

Job Queue

data source

1

convert the data into job and sent to queue

workerworkerworkerworkerworker

2

pull job and consume it

Job Queue

data source

1

convert the data into job and sent to queue

workerworkerworkerworkerworker

2

pull job and consume it

Result Queue

3

send result to another queue

Job Queue

data source

1

convert the data into job and sent to queue

workerworkerworkerworkerworker

2

pull job and consume it

Result Queue

3

send result to another queue

4

feed back to data

aggregator

pull result

Job Queue

data source

1

convert the data into job and sent to queue

workerworkerworkerworkerworker

2

pull job and consume it

Result Queue

3

send result to another queue

4

aggregator

pull result

make report

Example for video resolution

Job Queue

video table

{id, url, how-to-parse}

1

Job Queue1

workerworkerworkerworkerworker

2

pull job and consume it

video table

{id, url, how-to-parse}

Job Queue1

workerworkerworkerworkerworker

2

pull job and consume it

Result Queue

3

{id, width, height, codec}

video table

{id, url, how-to-parse}

Job Queue1

workerworkerworkerworkerworker

2

pull job and consume it

Result Queue

3

update video where id=? set width=?, height=?, codec=?

video table

{id, url, how-to-parse}

{id, width, height, codec}

4

aggregator

pull result

Example for streaming alive

Job Queue

streaming table

{id, url}

1

Job Queue1

workerworkerworkerworkerworker

2

pull job and consume it

streaming table

{id, url}

Job Queue1

workerworkerworkerworkerworker

2

pull job and consume it

Result Queue

3

{id, is_alive}

streaming table

{id, url}

Job Queue1

workerworkerworkerworkerworker

2

pull job and consume it

Result Queue

3

update streaming_status where id=? set status=?

4

aggregator

pull result

streaming table

{id, url}

{id, is_alive}

Deployment

Job Queue

workerworkerworkerworkerworkerResult Queue

aggregator

Everything can deploy locally, But

Job Queue

workerworkerworkerworkerworkerResult Queue

aggregator

Deploy Workers on the Cloudlet it more scalable