Upload
giuseppe-maxia
View
6.025
Download
1
Tags:
Embed Size (px)
DESCRIPTION
With Gearman, a client/server infrastructure for generic tasks, you can work on distributed servers easily, with little worry about the details.No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more.
Citation preview
Gearman
Sunday, 25 October 2009
Gearman: a technology for distributed
computingGiuseppe Maxia
MySQL Community Team Lead Sun Microsystems
Sunday, 25 October 2009
Mainframe
Mainframe
operating system
hardware
application
terminal
client
USER FRIENDLINESS
0 100
terminal
terminal
terminal
Sunday, 25 October 2009
Mini computers
Mainframe
operating system
hardware
application
client
USER FRIENDLINESS
0 100
Mini terminal
terminal
terminal
terminal
Mini
Sunday, 25 October 2009
Networks
server
hardware
application
client
USER FRIENDLINESS
0 100
personal computer
personal computer
personal computer
personal computer
hardware
operating system
operating system
hardware
Sunday, 25 October 2009
Web applications
web server
hardware
application
client
USER FRIENDLINESS
0 100
browser
browser
browser
browser
operating system
hardware
operating system
operating system
hardware
operating system
INT
ERN
ET
Sunday, 25 October 2009
Cloud applications
web server
hardware
application
client
USER FRIENDLINESS
0 100
browser
browser
browser
browser
operating system
hardware
operating system
operating system
hardware
service provider
applicationapplication
service providerservice
providerservice provider IN
TER
NET
Sunday, 25 October 2009
Some actors
• memcached
• gearman
Used in production by Facebook and Digg
Sunday, 25 October 2009
MAG ANE R
Sunday, 25 October 2009
MG ANER!=
Sunday, 25 October 2009
MAG ANE R ?
Sunday, 25 October 2009
M A GA N E R
Sunday, 25 October 2009
server worker client
taskjob
request
http://gearman.orgSunday, 25 October 2009
Distributed
Sunday, 25 October 2009
Multiple operating systems
Sunday, 25 October 2009
multiple languages
Sunday, 25 October 2009
freedom of choice
Sunday, 25 October 2009
redundancy
Sunday, 25 October 2009
USING GEARMAN
• Server: gearmand
• Client libraries:
• C/C++
• Java
• Perl
• PHP
• Python
Sunday, 25 October 2009
Simple usage GEARMAN
• Command line client and worker
Sunday, 25 October 2009
starting the server/usr/local/sbin/gearmand -d
# started as daemon.# No feedback given on the command line
Sunday, 25 October 2009
starting the server (2)/usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6)
# started as normal application# verbose output requested
Sunday, 25 October 2009
starting the workergearman -w -h hostname -p 4730 \ -f conta wc
# -w = act as worker# -f = function# conta = function name# wc = command to execute when function # 'conta' is called
Sunday, 25 October 2009
what the server says/usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) …
INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected
Sunday, 25 October 2009
starting the clientgearman -h hostname -p 4730 \ -f conta < ~/.bashrc 57 135 2149 # <- output # from worker
# -f = function# conta = function name# < ~/.bashrc = input data
Sunday, 25 October 2009
what the server says/usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) …
INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected … INFO Accepted connection from 127.0.0.1:5181 INFO [ 0] 127.0.0.1:5181 Connected INFO [ 0] 127.0.0.1:5181 Disconnected
Sunday, 25 October 2009
What happened
1 server start
listen to port 4730
Sunday, 25 October 2009
What happened
2 worker starts
registers function 'conta' to server
Sunday, 25 October 2009
What happened
3 client starts
requires function 'conta' from server
provides input data
Sunday, 25 October 2009
What happened
4 server sends client request to worker
passes all input data to worker
Sunday, 25 October 2009
What happened
5 worker receives request and data
processes input
Sunday, 25 October 2009
What happened
6 worker returns processed data
server passes it to client
Sunday, 25 October 2009
What happened
7 client receives processed data
client displays result
Sunday, 25 October 2009
A simple Perl worker
Sunday, 25 October 2009
A simple perl worker
1. add server
2. add function
3. loop
4. function definition
Sunday, 25 October 2009
simple worker (1)use strict;use warnings;use Gearman::XS qw(:constants);use Gearman::XS::Worker;
my $host = '127.0.0.1';my $port = 4730;
my $worker = new Gearman::XS::Worker;
my $ret = $worker->add_server($host, $port);if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $worker->error()); exit(1);}
Sunday, 25 October 2009
simple worker (2)my $options = '';
$ret = $worker->add_function( "reverse", # public function name 0, # timeout \&myreverse, # reference to function $options); # function argumentsif ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $worker->error());}
Sunday, 25 October 2009
simple worker (3)while (1) { my $ret = $worker->work(); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $worker->error()); }}
Sunday, 25 October 2009
simple worker (4)sub myreverse { my ($job) = @_;
my $workload = $job->workload(); my $result = reverse($workload);
printf("Job=%s F_Name=%s Workload=%s Result=%s\n", $job->handle(), $job->function_name(), $job->workload(), $result); return $result;}
Sunday, 25 October 2009
A simple Perl client
Sunday, 25 October 2009
A simple perl client
• add server
• run a task
Sunday, 25 October 2009
simple client (1)use strict;use warnings;use Gearman::XS qw(:constants);use Gearman::XS::Client;
my $client = new Gearman::XS::Client;
my $host = '127.0.0.1';my $port = 4730;my $ret = $client->add_server($host, $port);if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $client->error()); exit(1);}
Sunday, 25 October 2009
simple client (2)my $input = shift || 'teststring';
my ($return, $result) = $client->do("reverse", $input);if ($return == GEARMAN_SUCCESS) { printf("Result=%s\n", $result);}
Sunday, 25 October 2009
A sample run
Sunday, 25 October 2009
host 1perl worker.pl
Sunday, 25 October 2009
host 2perl client.plResult=gnirtstset
Sunday, 25 October 2009
host 1perl worker.plJob=H:gmac3.local:4 F_Name=reverse Workload=teststring Result=gnirtstset
Sunday, 25 October 2009
more client functions
• do_background
• add_task
• run_tasks
Sunday, 25 October 2009
Some advanced usage
• DBIx::SQLCrosstab
• Data cubes
• Perl only
Sunday, 25 October 2009
Sunday, 25 October 2009
Image processing
• CPU intensive
• Large storage needed
• Application is OS specific
Sunday, 25 October 2009
Sunday, 25 October 2009
Sunday, 25 October 2009
Remote sandboxes
Using GEARMANclient server architectureN serversN workersN clientsSupports multiple languages
Sunday, 25 October 2009
host4host3
host1 host2worker1 worker2
worker4
func1func2
worker3
worker5
func1
func1func3
func3
func1func3
func1func3
worker0func1func3
client
Gearmanserver
Gearmanserver
Sunday, 25 October 2009
host4host3
host1 host2worker1 worker2
worker4
func1func2
worker3
worker5
func1
func1func3
func3
func1func3
func1func3
worker0func1func3
Master
Slave2
Slave1
client
Gearmanserver
Gearmanserver
Gearmanserver
Sunday, 25 October 2009
Remote sandbox worker
•add server
•add function
•loop
•function definition
Sunday, 25 October 2009
remote sandbox worker (2)$ret = $worker->add_function( "make_sandbox", 0, \&make_sandbox, ''); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%s\n", $worker->error());}
Sunday, 25 October 2009
remote sandbox worker (4)sub make_sandbox { my ($job, $options) = @_;
my $command = $job->workload(); chomp $command; print STDERR "<$command>\n"; my $result = qx(make_sandbox $command) ; if ($?) { print $result, "\n"; return "Error ($!)\n"; } else { return "ok\n" }}
Sunday, 25 October 2009
Client to install Remote sandboxes
•add server
•run remote installation on master
•run remote installation on each slave
•run configuration queries on slaves
Sunday, 25 October 2009
sample call to remote workermy $ret = qx(echo "$MASTER_SANDBOX" | gearman -h $master_ip -f make_sandbox ) ;
# using the gearman command line utility
Sunday, 25 October 2009
More on similar matters
• FOSS.MY, Kuala Lumpur, Malaysia, 25-Oct-2009
• CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009
http://datacharmer.blogspot.com
http://gearman.orgSunday, 25 October 2009
THANKS
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.
Let's talk!
Sunday, 25 October 2009