Embed Size (px)
Citation preview
Clustersand where to find
them
Eugene Pirogovgmile
Nebo15Were hiring!
DatabasesTools
Theory
Takeaways
DatabasesTools
Theory
Takeaways
What isa cluster?
set of loosely or tightly connected computers that work
together so that, in many respects, they can be viewed as
a single system
When to usea cluster?
1. Fail-overclusters
2. Load balancingclusters
What typicalErlang clusteris built with?
1. A node
node()
2. An RPC call
:rpc.call(:nodex, M, :f, [“a”])
3. send call
send({MyProcess, :mynode}, :msg)
Example 1:Starting a node
iex
~> iexErlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)iex(1)> node():nonode@nohostiex(2)>
iex --name eugene
~> iex --name eugeneErlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)iex([email protected])1>
iex --sname eugene
~> iex --sname eugeneErlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)iex(eugene@Eugenes-MacBook-Pro-2)1>
iex --name eugene@host
~> iex --name eugene@hostErlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)iex(eugene@host)1>
Example 2:starting two nodes
iex --name [email protected] --name [email protected]
~> iex --name [email protected]([email protected])1>
~> iex --name [email protected]([email protected])1>
# On node1iex(1)> :net_adm.ping(:’[email protected]’):pong
Example 3:sending a message
iex --name node1iex --name node2
# On node2iex(1)> Process.register(Terminal, self())
# On node1iex(1)> send({Terminal, :’[email protected]’}, “Hello!”)
# On node2iex(2)> flush()“Hello!”
Example 4:calling remotely
# On node1iex([email protected])1> :rpc.call(:'[email protected]', Enum, :reverse, [100..1])[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, …]iex([email protected])2>
REST/JSON/XMLBinary protocol
REST/JSON/XMLBinary protocol
REST/JSON/XMLBinary protocol
Bonus track:Magic cookie!
cat ~/.erlang.cookie
iex --name node1 --cookie abciex --name node2 --cookie xyz
# On node1iex(1)> :erlang.get_cookie():abc
# On node2iex(1)> :erlang.get_cookie():xyz
# On node1iex(2)> :net_kernel.connect(:'[email protected]')false
# On node1iex(3)> :erlang.set_cookie(:’[email protected]’, :xyz)true
# On node1iex(4)> :net_kernel.connect(:'[email protected]')true
DatabasesTools
Theory
Takeaways
epmd
Erlang PortMapper Daemon
runs on system startup
~> ps ax | grep epmd25502 ?? S 0:11.53 /usr/local/Cellar/erlang/19.1/lib/erlang/erts-8.1/bin/epmd -daemon
maps portsto node names
net_kernel
Example 5:starting a
distributed node
iex
~> iexErlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)iex(1)>
~> iexErlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)iex(1)> node():nonode@nohostiex(2)>
~> iexErlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)iex(1)> node():nonode@nohostiex(2)> Process.registered() |> Enum.find(&(&1 == :net_kernel))nil
~> iexErlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)iex(1)> node():nonode@nohostiex(2)> Process.registered() |> Enum.find(&(&1 == :net_kernel))niliex(3)> :net_kernel.start([:’[email protected]’]){:ok, #PID<0.84.0>}iex([email protected])4>
~> iexErlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)iex(1)> node():nonode@nohostiex(2)> Process.registered() |> Enum.find(&(&1 == :net_kernel))niliex(3)> :net_kernel.start([:’[email protected]’]){:ok, #PID<0.84.0>}iex([email protected])4> Process.registered() |> Enum.find(&(&1 == :net_kernel)):net_kernel
Example 6:monitoring a
node
iex --name [email protected] --name [email protected]
iex([email protected])1>
iex([email protected])1>
iex([email protected])2> :net_kernel.monitor_nodes(true):ok
iex([email protected])3> :net_kernel.connect(:'[email protected]')true
iex([email protected])1>
iex([email protected])2> :net_kernel.monitor_nodes(true):ok
iex([email protected])3> :net_kernel.connect(:'[email protected]')true
iex([email protected])4> flush(){:nodeup, :"[email protected]"}:ok
iex([email protected])1>
iex([email protected])2> :net_kernel.monitor_nodes(true):ok
iex([email protected])3> :net_kernel.connect(:'[email protected]')true
iex([email protected])4> flush(){:nodeup, :"[email protected]"}:ok
# Ctrl+C on node2
iex([email protected])1>
iex([email protected])2> :net_kernel.monitor_nodes(true):ok
iex([email protected])3> :net_kernel.connect(:'[email protected]')true
iex([email protected])4> flush(){:nodeup, :"[email protected]"}:ok
# Ctrl+C on node2
iex([email protected])5> flush(){:nodedown, :"[email protected]"}:ok
iex([email protected])5>
net_adm
Example 7:ping
iex --name [email protected] --name [email protected]
iex([email protected])1>
iex([email protected])1>
iex([email protected])2> :net_adm.ping(:'[email protected]')pang
iex([email protected])2> :net_adm.ping(:'[email protected]')pong
Example 8:names
iex --name [email protected] --name [email protected]
iex([email protected])1>
iex([email protected])1>
iex([email protected])2> :net_adm.names(){:ok, [{'rabbit', 25672}, {'node1', 51813}, {'node2', 51815}]}
iex([email protected])3>
iex([email protected])1>
iex([email protected])2> :net_adm.names(){:ok, [{'rabbit', 25672}, {'node1', 51813}, {'node2', 51815}]}
iex([email protected])3> Node.list()[]
iex([email protected])4>
Example 9:world
# /etc/hosts127.0.0.1 host1.com127.0.0.1 host2.com127.0.0.1 host3.com
# /Users/gmile/.hosts.erlanghost1.com.host2.com.host3.com.
iex --name [email protected] --name [email protected] --name [email protected] --name [email protected] --name [email protected] --name [email protected]
iex([email protected])1>
iex([email protected])1>
iex([email protected])1> :net_adm.world()[:"[email protected]", :"[email protected]", :"[email protected]", :”[email protected]", :"[email protected]", :"[email protected]"]
iex([email protected])2>
Bonus track:Connecting
to a node running in production
iex --name [email protected] --cookie abc
$ iex --remsh [email protected] --cookie abc --name bar@localhostErlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)iex([email protected])1>
slave
3. Transfer configurationto slave nodes
2. Add code path to slave nodes
4. Ensure appsare started on slave
1. Start slave
What else?
Node
phoenixframework/firenest
bitwalker/swarm
Easy clustering, registration, and distribution of worker processes for
Erlang/Elixir
…a simple case where workers are dynamically created in response to some events under a supervisor, and
we want them to be distributed across the cluster and be discoverable by name from anywhere in the cluster
bitwalker/libcluster
What next?
…I didn’t want to resort to something like Docker,
because I wanted to see how far I could push Elixir and its
tooling.
DatabasesTools
Theory
Takeaways
mnesia
Example 10:initialize mnesia
iex --name [email protected] --name [email protected] --name [email protected]
iex([email protected])1> :mnesia.create_schema([:'[email protected]']):ok
iex([email protected])1> :mnesia.create_schema([:'[email protected]']):okiex([email protected])2> :mnesia.start():ok
iex([email protected])1> :mnesia.create_schema([:'[email protected]']):okiex([email protected])2> :mnesia.start():okiex([email protected])3> :mnesia.info()---> Processes holding locks <------> Processes waiting for locks <------> Participant transactions <------> Coordinator transactions <------> Uncertain transactions <------> Active tables <---schema : with 1 records occupying 413 words of mem===> System info in version "4.14.1", debug level = none <===opt_disc. Directory "/Users/gmile/[email protected]" is used.use fallback at restart = falserunning db nodes = ['[email protected]']stopped db nodes = []master node tables = []remote = []ram_copies = []disc_copies = [schema]disc_only_copies = [][{'[email protected]',disc_copies}] = [schema]2 transactions committed, 0 aborted, 0 restarted, 0 logged to disc0 held locks, 0 in queue; 0 local transactions, 0 remote0 transactions waits for other nodes: []:okiex([email protected])4>
iex([email protected])1> :mnesia.create_schema([:'[email protected]']):okiex([email protected])2> :mnesia.start():okiex([email protected])3> :mnesia.info()---> Processes holding locks <------> Processes waiting for locks <------> Participant transactions <------> Coordinator transactions <------> Uncertain transactions <------> Active tables <---schema : with 1 records occupying 413 words of mem===> System info in version "4.14.1", debug level = none <===opt_disc. Directory "/Users/gmile/[email protected]" is used.use fallback at restart = falserunning db nodes = ['[email protected]']stopped db nodes = []master node tables = []remote = []ram_copies = []disc_copies = [schema]disc_only_copies = [][{'[email protected]',disc_copies}] = [schema]2 transactions committed, 0 aborted, 0 restarted, 0 logged to disc0 held locks, 0 in queue; 0 local transactions, 0 remote0 transactions waits for other nodes: []:okiex([email protected])4>
“schema” table existsas a disk_copy (RAM + disk)
iex([email protected])2> :mnesia.start():ok
iex([email protected])2> :mnesia.start():ok
iex([email protected])2> :mnesia.start():ok
iex([email protected])2> :mnesia.change_config(:extra_db_nodes, [:’[email protected]’]){:ok, [:"[email protected]"]}
iex([email protected])2> :mnesia.start():ok
iex([email protected])2> :mnesia.start():ok
iex([email protected])2> :mnesia.change_config(:extra_db_nodes, [:’[email protected]’]){:ok, [:"[email protected]"]}
iex([email protected])3> :mnesia.change_config(:extra_db_nodes, [:’[email protected]’]){:ok, [:"[email protected]"]}
iex([email protected])2> :mnesia.start():ok
iex([email protected])2> :mnesia.start():ok
iex([email protected])2> :mnesia.change_config(:extra_db_nodes, [:’[email protected]’]){:ok, [:"[email protected]"]}
iex([email protected])3> :mnesia.change_config(:extra_db_nodes, [:’[email protected]’]){:ok, [:"[email protected]"]}
iex([email protected])1> :mnesia.create_table(:books, [disc_copies: [:'[email protected]'], attributes: [:id, :title, :year]]):ok
iex([email protected])2> :mnesia.start():ok
iex([email protected])2> :mnesia.start():ok
iex([email protected])2> :mnesia.change_config(:extra_db_nodes, [:’[email protected]’]){:ok, [:"[email protected]"]}
iex([email protected])3> :mnesia.change_config(:extra_db_nodes, [:’[email protected]’]){:ok, [:"[email protected]"]}
iex([email protected])1> :mnesia.create_table(:books, [disc_copies: [:'[email protected]'], attributes: [:id, :title, :year]]):ok
iex([email protected])4> :mnesia.add_table_copy(:books, :'[email protected]', :ram_copies):ok
iex([email protected])2> :mnesia.start():ok
iex([email protected])2> :mnesia.start():ok
iex([email protected])2> :mnesia.change_config(:extra_db_nodes, [:’[email protected]’]){:ok, [:"[email protected]"]}
iex([email protected])3> :mnesia.change_config(:extra_db_nodes, [:’[email protected]’]){:ok, [:"[email protected]"]}
iex([email protected])1> :mnesia.create_table(:books, [disc_copies: [:'[email protected]'], attributes: [:id, :title, :year]]):ok
iex([email protected])4> :mnesia.add_table_copy(:books, :'[email protected]', :ram_copies):ok
iex([email protected])5> :mnesia.add_table_copy(:books, :'[email protected]', :ram_copies):ok
iex([email protected])6> :mnesia.info()---> Processes holding locks <------> Processes waiting for locks <------> Participant transactions <------> Coordinator transactions <------> Uncertain transactions <------> Active tables <---books : with 0 records occupying 304 words of memschema : with 2 records occupying 566 words of mem===> System info in version "4.14.1", debug level = none <===opt_disc. Directory "/Users/gmile/[email protected]" is used.use fallback at restart = falserunning db nodes = ['[email protected]','[email protected]','[email protected]']stopped db nodes = []master node tables = []remote = []ram_copies = []disc_copies = [books,schema]disc_only_copies = [][{'[email protected]',disc_copies}, {'[email protected]',ram_copies}, {'[email protected]',ram_copies}] = [schema,books]12 transactions committed, 0 aborted, 0 restarted, 10 logged to disc0 held locks, 0 in queue; 0 local transactions, 0 remote0 transactions waits for other nodes: []:ok
iex([email protected])32>
iex([email protected])6> :mnesia.info()---> Processes holding locks <------> Processes waiting for locks <------> Participant transactions <------> Coordinator transactions <------> Uncertain transactions <------> Active tables <---books : with 0 records occupying 304 words of memschema : with 2 records occupying 566 words of mem===> System info in version "4.14.1", debug level = none <===opt_disc. Directory "/Users/gmile/[email protected]" is used.use fallback at restart = falserunning db nodes = ['[email protected]','[email protected]','[email protected]']stopped db nodes = []master node tables = []remote = []ram_copies = []disc_copies = [books,schema]disc_only_copies = [][{'[email protected]',disc_copies}, {'[email protected]',ram_copies}, {'[email protected]',ram_copies}] = [schema,books]12 transactions committed, 0 aborted, 0 restarted, 10 logged to disc0 held locks, 0 in queue; 0 local transactions, 0 remote0 transactions waits for other nodes: []:ok
iex([email protected])32>
“schema” + “books” tables existon 3 different nodes
3 nodes are running
current node (node1)keeps 2 tables as RAM + disk
Before we proceed…
CAP theorem!
@b0rk
Consistency
Every read receives the most recent write or an error
Availability
Every request receives a response, without guarantee that
it contains the most recent version of the information
Partition tolerance
The system continues to operate despite an arbitrary number of messages being dropped by the network between nodes
Pick two!AP or AC or CP
ACis kind ofpointless
Mnesia chooses…
AC!
If in your cluster the network connection between two nodes
goes bad, then each one will think that the other node is down,
and continue to write data.
Recovery from this is complicated.
AXD 301switch
“…measures are taken such that network reliability is very high”
“…In such a highly specialized environment, the reliability of the control backplane essentially removes some of
the worries which the CAP theorem introduces.”
DatabasesTools
Theory
Takeaways
1. Elixir doesn’t bring anything new
to the table…
…apart from productivity!
It’s allabout Erlang
2. “Hello world” for clusters is simple
3. Deciding what matters is hard
Understahd your values when
building a cluster!
4. Releasing & deploying stuff is
tricky
5. An Erlang app can be your little
universe
