dtools documentation

8/8/2019 dtools documentation

1/23

dtools DocumentationRelease 4.0

Andrs J. Daz

November 17, 2010


2/23


3/23

CONTENTS

1 Installation 3

1.1 Gentoo & Exherbo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 First Contact 52.1 Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3 The dtools database 7

3.1 Manage the database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4 Patterns 9

4.1 System pattern (sys) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.2 Output pattern (out) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.3 Expression pattern (exp) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.4 Tag pattern (tag) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.5 Combining patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

5 Commands 11

5.1 The add command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

5.2 The ipmitool command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

5.3 The list command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.4 The ping command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.5 The rscp command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.6 The scp command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.7 The script command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.8 The ssh command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5.9 The ssh-copy-id command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5.10 The sudo command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5.11 The tag command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

6 Examples 15

6.1 Add a number of hosts manually . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

6.2 Bulk adding hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

6.3 Authorize hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

6.4 Ping all hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

7 For impatients 17

8 License 19

i


4/23


5/23

dtools Documentation, Release 4.0

Author Andres J. Diaz

Abstract dtools is a suite of utilities written in pure bash to handle a number of nodes in a distributed

network, its similar to other distributed shells, like dsh or pyDSH, but designed to be modular

and easy-to-extend.

CONTENTS 1
mailto:[email protected]://www.netfort.gr.jp/~dancer/software/dsh.html.enhttp://pydsh.sourceforge.net/http://pydsh.sourceforge.net/http://www.netfort.gr.jp/~dancer/software/dsh.html.enmailto:[email protected]


6/23


2 CONTENTS


7/23

CHAPTER

ONE

INSTALLATION

Here are the installation instructions for dtools, our goal is to provide a custom installation for each GNU/Linux

and UNIX distributions, in the meanwhile, a generic overview is described.

First you can download the currently working branch from github main repository using git protocol:

$ git clone git://github.com/ajdiaz/dtools

Or even download an specified version:

$ wget -O dtools-4.0.tar.gz http://github.com/ajdiaz/dtools/tarball/4.0

Then, once unpacked, you can run dtools directly from the working directory, prepending the DTOOLS_LIB

variable to change the library path:

$ cd dtools-4.0

$ DTOOLS_LIB=lib/ ./dt -h

Or you can copy these files to the system as root:

# cp dt dthost dtstatus /usr/bin

# cp -r lib /usr/lib/dtools

# cp -r doc/man/\*.1 /usr/share/man/man1

Also you have packages for a number of GNU/Linux distributions

1.1 Gentoo & Exherbo

Please use the paludis repository ajdiaz-exheres, and then:

# paludis -i dtools

Note: If you like to contribute with any package, please contact me to devel atlists.connectical.com

3
http://github.com/ajdiaz/ajdiaz-exhereshttp://github.com/ajdiaz/ajdiaz-exheres


8/23


4 Chapter 1. Installation


9/23

CHAPTER

TWO

FIRST CONTACT

This is the main syntax for dt invokation:

dt [options] : [args]

The options can modify the normal behaviour of dt, including the dtools database location, the number of paralell

executions and more.

The pattern is a way to get the targets. The targets are the nodes in database where a command will be executed.

Usually the pattern get a subset of nodes in database.

The command is the action which will be executed in the remote nodes affected by the pattern matching. Note

that the could be not just a shell command, but a dtools command, for example the ssh command run using ssh

protocol a number of commands which will be passed in args.

Note: Please read the section Examples for a number of well explained examples, which is the more usefull part

of this documentation too.

2.1 OptionsThe dtools script accept the following options:

-h X --help X display help about X command or this help screen.

-D X --db use X as known_hosts database.

-p --pretend pretend only and do nothing.

-C X --coproc X use X coprocs (default:5).

-T X --timeout X use X seconds of timeout (default:5).

-i --interactive interactive, an alias to -C 0

-N --no-color supress color output

2.2 Features

Modular, so its easy to extend, just only need to create your own pattern or command file.

Thinking in sysadmin. No development skills requirement to extend the software.

Fast. Using coprocs to run a number of hosts in paralel.

Easy to parse. The output format is a colon-separated list, which is easy to parse for non-humans, but

easy-to-read for humans too.

Easy to deploy. Written in bash, just only need a modern shell to run it.

Tested. Used in production environment for years, with more of 400 hosts.

5


10/23


6 Chapter 2. First Contact


11/23


12/23


8 Chapter 3. The dtools database


13/23

CHAPTER

FOUR

PATTERNS

As we explained in Introduction, the patterns are a way to filter the dtools database to find a properly target set.

You can create your own patterns as described in Developmentsection, but dtools has a number of them for you.

4.1 System pattern (sys)

Description Acts over hostname passed in pattern syntax.

Syntax sys:

Example sys:host1.connectical.com

4.2 Output pattern (out)

Description Get the database from the stdin. Usefull to combine with other patterns (see Combin-

ing patterns section for more information).Syntax out:

Example out::

4.3 Expression pattern (exp)

Description Filter the database using a regular expression matching for host.

Syntax exp:

Example exp:db.*

4.4 Tag pattern (tag)

Description Get from the database the hosts which have the specified tag operation.

Syntax tag:

Example tag:database

OR Operation

Operator ,

Meaning return hosts which match which almost one of two elements separated by the operator.Examples tag:database,webserver

9


14/23


AND Operation

Operator +

Meaning return hosts which match with both tags.

Examples tag:database+oracle

EXCEPT Operation

Operator -

Meaning returng hosts which match for the first tag, but not for the second one.

Examples tag:database-oracle

4.5 Combining patterns

You can filter the result of one pattern with other pattern. To do this just append two-colons and the next pattern

to filter. For example, lets suppose that you can filter your database using a tag pattern, i.e. all nodes with tag

mysql, and then you can filter this result with nodes which name starts with db, then you can do something similarto this:

tag:mysql:exp:db.*

The pattern combination are a powerfull tool to play with database filters, neverthless, in practice, one pattern

would be enough for every day use.

An usefull case of pattern combination is the out pattern, with the out pattern you can read hosts from stdin

and filter them with normal patterns, just in a combination form. For example:

out:tag:mysql

10 Chapter 4. Patterns


15/23

CHAPTER

FIVE

COMMANDS

The commands are the actions that dtools can performs. Be carefull with commands, when we talk about dtools

commands we are not talking about running commands on remote systems, in fact, run a remote command using

SSH is just only dtool command. To avoid this problem we callactions to the dtools commands, leaving commands

for the remote commands than we perform over SSH or other protocol.

There are a list of currently available commands in dtool:

add add host to dtools database

ipmitool manage IPMI interface on remote hosts

list list hosts

ping ping hosts

rscp reverse scp from host to local machine

scp classic scp copy to hosts

script execute a local script in remote hosts

ssh execute a command in remote hosts

ssh-copy-id distribute a public key in hosts

sudo execute a command in remote hosts with privilegies

tag manage tagging features

5.1 The add command

Syntax add [tags]+

Meaning add the affected hosts to the host database, tagging them if any tag is supplied.

Context local

Requirements None

Example dt sys:mynewhost add

5.2 The ipmitool command

Syntax ipmitool [ipmitool_options]

Meaning perform IPMI actions to the affected hosts using ipmitool

Context remote

Requirements ipmitool

Example dt tag:all ipmitool cycle

11


16/23


5.3 The list command

Syntax list

Meaning list affected hosts

Context local

Requirements None

Example dt tag:all list

5.4 The ping command

Syntax ping

Meaning perform a classic ICMP ping to the affected host, and measure the trip time.

Context remote

Requirements ping

Example dt tag:all ping

5.5 The rscp command

Syntax rscp

Meaning reverse copy (from remote host to local host), using SCP.

Context remote

Requirements scp

Example dt tag:all rscp /etc/passwd /tmp/override_passwd

5.6 The scp command

Syntax scp

Meaning direct copy (from local host to remote host) using SCP

Context remote

Requirements scp

Example dt tag:all scp /etc/passwd /etc/passwd

5.7 The script command

Syntax script

Meaning run in the remote host the script which is located in local host.

Context remote

Requirements ssh

Example dt tag:all script ./create_users.sh

12 Chapter 5. Commands


17/23


5.8 The ssh command

Syntax ssh [ssh_options]

Meaning run in the remote host the arbitrary action using SSH

Context remote

Requirements ssh

Example dt tag:all ssh date

5.9 The ssh-copy-id command

Syntax ssh-copy-id

Meaning distributed a pubkey to affected hosts.

Context remote

Requirements ssh-copy-id

Example dt tag:all ssh-copy-id ~/.ssh/id_rsa.pub

5.10 The sudo command

Syntax sudo [sudo_options]

Meaning perform a remote action with superuser permissions with sudo

Context remote

Requirements ssh (local), sudo (remote)

Example dt tag:all sudo deluser testing

5.11 The tag command

Syntax tag [ +, - ]+

Meaning add or remove tags to the affected hosts.

Context local

Requirements None

Example dt tag:all tag +everyhost

5.8. The ssh command 13


18/23


14 Chapter 5. Commands


19/23

CHAPTER

SIX

EXAMPLES

Here are a number of examples using dtools, from the basic use to more advanced use.

6.1 Add a number of hosts manually

$ dt sys:myhost1.com add

okay::dt:add:myhost1.com:

$ dt sys:myhost2.com add


...

6.2 Bulk adding hosts

Lets suppose that host are in a file called myhosts.txt, one host per line:

$ cat myhosts.txtmyhost1.com

myhost2.com

$ cat myhosts.txt | dt out:: add



6.3 Authorize hosts

Once hosts are added to dtools database, you might want authorize them. When a host is authorized, you can use

SSH commands without need to type passwords, using public key handshake.

$ ssh-keygen

Generating public/private rsa key pair.

Enter file in which to save the key (~/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in asd.

Your public key has been saved in asd.pub.

The key fingerprint is:

cf:19:ce:44:45:35:78:11:ce:04:23:eb:fb:e9:9f:cc ajdiaz@boole

The keys randomart image is:

+--[ RSA 2048]----+

| ..=+*o |

| +.=.. |

| o .o |

| o |

15


20/23


| S + |

| * + |

| B |

| . + . |

| .+.E |

+-----------------+

$ dt exp:.* ssh-copy-id ~/.ssh/id_rsaokay::dt:ssh-copy-id:myhost1.com:

okay::dt:ssh-copy-id:myhost1.com:

6.4 Ping all hosts

$ dt exp:.* ping

okay::dt:ping:myhost1.com:0ms

okay::dt:ping:myhost1.com:0ms

16 Chapter 6. Examples


21/23

CHAPTER

SEVEN

FOR IMPATIENTS

First add hosts to database

$ dt sys:myhost.com add

$ dt sys:otherhost.net add

Then send your public key for ssh authentication

$ dt exp:.* ssh-copy-id ~/.ssh/id_dsa.pub

Finally, enjoy!

$ dt exp:.* list

17


22/23


18 Chapter 7. For impatients


23/23

CHAPTER

EIGHT

LICENSE

This code is releasing under GPLv2 license.
http://www.gnu.org/licenses/gpl-2.0-standalone.htmlhttp://www.gnu.org/licenses/gpl-2.0-standalone.html

Documents

dtools documentation