HIL Documentation - Read the DocsHIL Documentation, Release 1.0 1.1Requirements Required software/hardware for running a production HIL include: •Network switches: – At least one

HIL DocumentationRelease 1.0

HIL Team

Dec 04, 2018

Contents

1 HIL - Introduction 3

2 HIL Architecture Overview 5

3 INSTALL 7

4 INSTALL-devel-tldr 17

5 INSTALL-devel 19

6 Upgrading HIL 25

7 Using HIL as a Client 29

8 Extensions 33

9 Overview 35

10 API Reference 37

11 Deployment 63

12 Consistency model 67

13 Part 1: Install PostgreSQL server. 69

14 Keystone Authentication 73

15 Logging 75

16 Introduction 77

17 Network Drivers 81

18 Types of Networks in the HIL 89

19 Network Teardown 91

20 Testing 93

i

21 VLAN Primer 97

22 Driver Model 101

23 Search 103

ii

HIL Documentation, Release 1.0

Contents:

Contents 1


2 Contents

CHAPTER 1

HIL - Introduction

HIL is a low-level tool for reserving physical machines and connecting them via isolated networks. It does notprescribe a particular method for imaging/managing said machines, allowing the user to use any solution of theirchoosing.

HIL keeps track of available resources in a database, which a system administrator must populate initially.

This includes information such as:

• What machines are available

• What network interfaces they have

• Where those NICs are connected (what port on what switch)

From there, a regular user may:

• Reserve physical machines

• Create isolated logical networks

• Create “headnodes,” which are small virtual machines usable for management/provisioning purposes

• Connect network interfaces belonging to physical and/or headnodes to logical networks.

• Reboot their machines, view the serial consoles – aditionaly such management features may exist in the future.

A typical user workflow might look like:

1. Reserve some machines.

2. Create a logical “provisioning” network.

3. Connect a NIC from each machine to the provisioning network. In particular, one could connect a NIC fromwhich the machine will attempt to boot.

4. Create a headnode, and attach it to the provisioning network

5. Log in to the headnode, set up a PXE server, reboot the nodes, and deploy an operating system on them via thenetwork.

3


1.1 Requirements

Required software/hardware for running a production HIL include:

• Network switches:

– At least one switch from the Cisco Nexus 5xxx or Dell PowerConnect 55xx families

– For environments including more than one switch, all VLANs must be trunked to all managed switches

• A single node that has the following:

– A webserver capable of supporting the WSGI standard (Apache/mod_wsgi is the only one tested)

– python 2.7, with the ability to install packages via pip

– Access to:

* The Internet or intranet (a way for users to connect to the HIL service)

* The administrative telnet IP on the managed switches

– Currently only CentOS and RHEL 7.x have been tested, though any node that otherwise meets theserequirements should function.

• Database: a Postgres database server. Sqlite works but is not recommended for production.

For IPMI proxy functionality : * Network access from the HIL service node to the IPMI interfaces of node undermanagement * Nodes that support IPMI v2+ * A recent version of ipmitool installed on the HIL service node

For headnode functionality:

• A recent Linux version for the HIL service node that has libvirt with KVM installed

• Some number of VM templates

• A trunk port connected between the switch and HIL service node that carries all VLANs accessible from HIL

1.2 Documentation

• The full documentation is availalbe at ReadTheDocs in a beautiful and easy to navigate web interface.

• The docs directory contains all the documentation in .rst and .md format

• Examples contains examples of config files, templates for creating headnode VM images and a script to registernodes with HIL.

1.3 Mass Open Cloud

This project is part of the larger Mass Open Cloud. For a description of the team and other information, see https://github.com/CCI-MOC/moc-public/blob/master/README.md.

4 Chapter 1. HIL - Introduction

http://hil.readthedocs.io/en/latest/

https://github.com/CCI-MOC/hil/tree/master/docs

https://github.com/CCI-MOC/hil/tree/master/examples

http://www.massopencloud.org

https://github.com/CCI-MOC/moc-public/blob/master/README.md

https://github.com/CCI-MOC/moc-public/blob/master/README.md

CHAPTER 2

HIL Architecture Overview

The HIL (Hardware Isolation Layer) is a tool used to perform network isolation of physical machines, so that differentparties can be given physical capacity within the same data center, without needing to trust each other.

2.1 Operation

From a user’s perspective, the HIL allows one to:

• allocate physical nodes

• create login/management nodes (sometimes called headnodes)

• configure networks, in particular:

– create logical networks

– connect physical nodes to those logical networks (on a per-nic basis)

• connect login/management nodes to those networks.

Right now, we’re using 802.1q VLANs to achieve network isolation. The HIL communicates with a managed switch,to which the physical hardware is attached. When networking operations are performed, the HIL sends commands tothe switch which configure the relevant ports as needed to create the logical networks.

2.2 Anatomy of a Running Installation

SWITCH_____________

<nic1>---------------] access{N} |=node-1= | |

<nic2>---------------] || || | (

→˓ ) (continues on next page)

5


(continued from previous page)

| | ( ^br-vlanM^------^trunk-nic.M^→˓ )<nic1>---------------] access{M} | ( %hn-B%

→˓ )=node-2= | | (→˓ )<nic2>---------------] | ( %hn-A%

→˓ )| | ( ^br-vlanN^------^trunk-nic.N^

→˓ )| | (

→˓ )| | (

→˓ )<nic1>---------------] trunk [--------<trunk-nic>=hil-master=

=node-3= | |<nic2>---------------]___________[

Legend:

( running in software )<physical network interface>^virtual/logical network interface^=physical node=%virtualized node%]/[ switch's Ethernet ports--- connection (virtual or physical)access{X} Denotes that the adjacent port is set to access mode, with vlan #X.trunk Denotes that the adjacent port is set to trunk mode.

A typical installation of the HIL will have the following components:

• The HIL API server and headnode VM host

• A managed switch

• One or more physical nodes, each of which has one or more network interfaces.

These components will be configured as follows:

• All of the physical nodes will have some subset (possibly all) of their nics connected to the managed switch.

– The ports that these are connected to will be set to access mode.

• The HIL headnode host will have one nic connected to the managed switch.

– The corresponding port will be set to trunk mode, with all vlans enabled.

• The HIL master will be running the libvirt daemon, which will have at least one VM, powered off, which canbe cloned and started to provide login/management nodes.

• A network object in the HIL corresponds to a vlan id. (In future versions, we will also allow other mechanisms,such as VXLAN.) Network operations have the following effects

– Ports added to the network will have their access vlan set to the vlan id associated with the network.

– Ports removed from the network will be set to access no vlans.

6 Chapter 2. HIL Architecture Overview

CHAPTER 3

INSTALL

This document describes the installation and setup of HIL on CentOS 7.0. HIL should work on other distros, but isnot well tested or supported. For development environments, see Installation - Developers.

3.1 The HIL Service node

This section talks about what must be done on the server upon which HIL runs.

3.2 Prerequisite Software

HIL requires a number of packages available through the CentOS RPM repositories, as well as the EPEL repository.EPEL can be enabled via:

$ yum install epel-release

Then, the rest of the packages can be installed via:

$ yum install libvirt bridge-utils ipmitool telnet httpd mod_wsgi python-pip qemu-kvm→˓python-virtinst virt-install python-psycopg2 vconfig net-tools

In addition, HIL depends on a number of python libraries. Many of these are available as RPMs as well, but werecommend installing them with pip, since this will install the versions that HIL has been tested with. This is doneautomatically by the instructions below.

3.3 Disable SELinux

The setup described in this document runs into problems with SELinux. In the future, we hope to ship a set of SELinuxsecurity policies with HIL, but for now the solution is to disable SELinux:

7

INSTALL-devel.html


$ sudo setenforce 0

Make sure SELinux is also disabled on startup. To do this on CentOS/RHEL, edit /etc/selinux/config to change: `SELINUX=enforcing ` to ` SELINUX=permissive `

3.4 User needs to choose appropriate values for their environment:

For simplicity we have provided default values: Copy the following lines in file hil_env

export HIL_USER=hil

export HIL_ADMIN=hil

export HIL_ADMIN_PASSWORD=secret

export HIL_HOME_DIR=/var/lib/hil

Before starting this procedure do:

$ source hil_env

3.5 Setting up system user for HIL:

First create a system user ${HIL_USER} with:

$ sudo useradd --system ${HIL_USER} -d /var/lib/hil -m -r

The HIL software itself can then be installed as root by running:

$ sudo su -$ cd /root$ git clone https://github.com/CCI-MOC/hil$ cd hil$ python setup.py install

3.6 hil.cfg

HIL is configured with hil.cfg. This file contains settings for both the CLI client and the server. If hil.cfg isnot found, hil would refuse to run and exit. Carefully read the hil.cfg* files in examples/, to understand andcorrectly set all of the options. In particular, the following two fields in the headnode section are very important:trunk_nic must match your choice of trunk NIC in the “Networking - Bridges” instructions below; base_imgsmust match the name of the base headnode libvirt instance created in the “Libvirt” instructions below.

For a detailed description of the configuration needed for various switch setups, see Network Drivers.

Logging level and directory can be set in the [general] section. For more information view logging.

hil.cfg file contains sensitive administrative information and should not be exposed to clients or end users. Ac-cordingly, HIL checks the permissions on startup and refuses to run if it is accessible by any user other than its owner.Ideally the file should also be read-only, as write access is not required at runtime. From the source directory of hil asuser root do the following:

8 Chapter 3. INSTALL

https://wiki.centos.org/HowTos/SELinux

network-drivers.html

logging.html


(from /root/hil)$ cp examples/hil.cfg /etc/hil.cfg$ chown ${HIL_USER}:${HIL_USER} hil.cfg$ chmod 400 hil.cfg(run following command as ${HIL_USER} from ${HIL_HOME_DIR}$ su - ${HIL_USER}$ ln -s /etc/hil.cfg .

3.7 Authentication and Authorization

HIL includes a pluggable architecture for authentication and authorization. HIL ships with two authentication back-ends. One uses HTTP basic auth, with usernames and passwords stored in the hil database. The other is a “null”backend, which does no authentication or authorization checks. This can be useful for testing and experimentation butshould not be used in production. You must enable exactly one auth backend.

In productions system where non-null backend is active, end users will have to include a username and passwordas additional parameters in client_env file to be able to communicate with the hil server. This is user/passwordshould be registered with the hil auth backend using hil.

3.7.1 Database Backend

To enable the database backend, make sure the [extensions] section of hil.cfg contains:

hil.ext.auth.database =

3.7.2 Keystone Backend

To enable the Keystone backend, make sure the [extensions] section of hil.cfg contains:

hil.ext.auth.keystone =

Visit the Keystone configuration guide for more information.

3.7.3 Null Backend

To enable the null backend, make sure [extensions] contains:

hil.ext.auth.null =

3.8 Setting Up HIL Database

The only DBMS currently supported for production use is PostgreSQL. (SQLite is supported for development purposesonly). There are many ways of setting up PostgreSQL server. Install configure PostgreSQL CENTOS7. provides oneway to accomplish this.

To create the database tables, first make sure hil.cfg is set up the way you need, including any extensions you planto use, then:

3.7. Authentication and Authorization 9

keystone-auth.html

Install_configure_PostgreSQL_CENTOS7.html


$ sudo -i -u ${HIL_USER}; hil-admin db create

If the authorization backend activated in hil.cfg is hil.ext.auth.database = then you will need to add aninitial user with administrative privileges to the database in order to bootstrap the system. You can do this by runningthe following command (as user hil):

$ sudo -i -u ${HIL_USER}; hil-admin create-admin-user ${HIL_ADMIN_USER} ${HIL_ADMIN_→˓PASSWORD}

You can then create additional users via the HTTP API. You may want to subsequently delete the initial user; this canalso be done via the API.

All HIL commands in these instructions should be run in this directory:

$ cd /var/lib/hil

3.9 Networking - Bridges

Currently HIL only supports one mechanism for layer-2 isolation: 802.1q VLANs. One NIC on the HIL host isdesignated the “trunk NIC”. All network traffic to headnode VMs in HIL is routed through this trunk NIC, on atagged VLAN. As such, the port on the switch that this NIC connects to must have all of HIL’s VLANs trunked to it.Currently, this configuration must be done manually.

HIL uses Linux bridges to route the traffic from the trunk NIC to the headnodes. Currently the bridges and VLANNICs for this must be created ahead of time. The provided script create_bridges will create bridges for allVLANS in the allocation pool. It must be run in the directory that contains hil.cfg. This pre-allocation is easierto reason about than on-demand creation, and allows HIL to be run as an unprivileged user, but it also causes somelimitations. For instance, because of this, headnodes can only be connected to networks with allocated VLANs. Thebridges must also be pre-allocated again on each boot. For now, the recommened method is to use systemd. Asystemd service for running the create_bridges script is available in the ‘scripts’ directory.

Name of the service is: create_bridges.service

Name of the script is: create_bridges

3.9.1 Centos:

Centos uses systemd to controll all its processes.

Place the file create_bridges.service under: /usr/lib/systemd/system/

3.9.2 Ubuntu:

Systemd is available from Ubuntu 15.04 onwards and LTS version 16.04 will ship with systemd by default.

Edit the create_bridges.service file and change the ExecStart to /usr/local/bin/create_bridges

Place the file create_bridges.service under: /lib/systemd/system/

3.9.3 Starting the service:

Following commands will start the daemon: $ systemctl daemon-reload $ systemctl startcreate_bridges.service



You can check the status using: $ systemctl status create_bridges.service

To auto-start the service on boot (recommended): $ systemctl enable create_bridges.service

3.9.4 For systems that do not support systemd:

You can add the following line:

($ cd /etc && create_bridges)

to the end of /etc/rc.local.

You can also run the this command manually as root user to create the bridges.

HIL must additionally have IP connectivity to the switch’s administration console. Right now the only mechanism forconnecting to the switch is via telnet (with plans to support ssh). As such, the administration console should only beaccessible through a trusted private network.

3.10 Libvirt

We must reconfigure libvirt to allow (some) unprivileged users access to the system QEMU session. To do this,edit /etc/libvirt/libvirtd.conf and uncomment the following lines:

unix_sock_group = "libvirt"auth_unix_ro = "none"auth_unix_rw = "none"

Then create the group ‘libvirt’ and add the HIL user to that group:

$ sudo groupadd libvirt$ sudo gpasswd libvirt -a hil

Finally, restart libvirt with:

$ sudo service libvirtd restart

You should also set libvirt to start on boot:

$ sudo chkconfig libvirtd on

3.10.1 Headnode image

Now we must make a clonable base headnode. (One is required, and more are allowed.) First create a storage pool.Any kind can be used, but we will only document creating a directory-backed storage pool:

virsh --connect qemu:///system pool-define pool.xml

where pool.xml contains a description of the pool:

<pool type="dir"><name>hil_headnodes</name><target><path>/var/lib/libvirt/images</path>

(continues on next page)

3.10. Libvirt 11

https://github.com/CCI-MOC/hil/issues/46



</target></pool>

The directory specified by path must already exist, and be readable and writable by the libvirt user. Then activatethe pool, and make the it activate on boot, with:

$ virsh --connect qemu:///system pool-start hil_headnodes$ virsh --connect qemu:///system pool-autostart hil_headnodes

The scripts in examples/cloud-img-with-passwd can be used to build an ubuntu 14.04 or centos 7 diskimage with a default root password. Read the README in that directory for more information.

Once the disk image is built, copy ito the storage pool directory (here we assume it is called base.img):

$ mv base.img /var/lib/libvirt/images/

Finally, create the base headnode with:

$ virsh --connect qemu:///system define base.xml

where base.xml contains a description of the headnode:

<domain type='kvm'><name>base</name><memory>524288</memory><os><type arch='x86_64'>hvm</type><boot dev='hd'/>

</os><features><acpi/><apic/><pae/>

</features><clock offset="utc"/><on_poweroff>destroy</on_poweroff><on_reboot>restart</on_reboot><on_crash>restart</on_crash><vcpu>1</vcpu><devices><emulator>/usr/libexec/qemu-kvm</emulator><disk type='file' device='disk'>

<driver name='qemu' type='raw'/><source file='/var/lib/libvirt/images/base.img'/><target dev='vda' bus='virtio'/>

</disk><interface type='network'>

<source network='default'/><model type='virtio'/>

</interface><input type='tablet' bus='usb'/><graphics type='vnc'/><console type='pty'/><sound model='ac97'/><video>

<model type='cirrus'/></video>

</devices></domain>



Note that the above specifies the format of the disk image as raw; if you’re using an image in another format (such asqcow) you will have to adjust this.

Many of these fields are probably not needed, but we have not thouroughly tested which ones. Furthermore, this set ofXML duplicates the path to storage directory; this seems unnecessary.

Users may find the scripts in examples/puppet_headnode useful for configuring the ubuntu headnode to act asa PXE server; see the README in that directory for more information.

3.11 Running the Server under Apache

HIL consists of two services: an API server and a networking server. The former is a WSGI application, whichwe recommend running with Apache’s mod_wsgi. Create a file /etc/httpd/conf.d/wsgi.conf, with thecontents:

LoadModule wsgi_module modules/mod_wsgi.soWSGISocketPrefix run/wsgi

<VirtualHost 127.0.0.1:80 [::1]:80>ServerName 127.0.0.1AllowEncodedSlashes OnWSGIPassAuthorization OnWSGIDaemonProcess hil user=hil group=hil threads=2WSGIScriptAlias / /var/www/hil/hil.wsgi<Directory /var/www/hil>WSGIProcessGroup hilWSGIApplicationGroup %{GLOBAL}Order deny,allowAllow from all

</Directory></VirtualHost>

(The file may already exist, with just the LoadModule option. If so, it is safe to replace it.)

Note: if accessing HIL through a public IP address, be sure to change the VirtualHost entry accordingly. Virtual-Host documentation

Note: certain calls to HIL such as port_register() may pass arbitrary strings that should be escaped (see issue 361).By default, Apache Doesn’t allow this due to security concerns. AllowEncodedSlashes On enables the passingof these arguments.

Note: For apache to be able to pass the authentication headers to HIL following directive will have to be turned on

WSGIPassAuthorization On

(see http://stackoverflow.com/questions/20940651/how-to-access-apache-basic-authentication-user-in-flask )

If you haven’t already, create the directory that will contain the HIL WSGI module:

$ sudo mkdir /var/www/hil/

Copy the file hil.wsgi from the top of the hil source tree to the location indicated by the WSGIScriptAliasoption. The virtual host and server name should be set according to the hostname (and port) by which clients willaccess the api. Then, restart Apache:

$ sudo service httpd restart

You should also set apache to start on boot:

3.11. Running the Server under Apache 13

https://httpd.apache.org/docs/current/mod/core.html#virtualhost

https://httpd.apache.org/docs/current/mod/core.html#virtualhost


https://stackoverflow.com/questions/4390436/need-to-allow-encoded-slashes-on-apache

http://stackoverflow.com/questions/20940651/how-to-access-apache-basic-authentication-user-in-flask


$ sudo chkconfig httpd on

Note:

It is recommended to use TLS when running HIL in production.

You can use [Let’s Encrypt](https://letsencrypt.org/) to generate free certificates. [Certbot](https://certbot.eff.org/)makes this process easier. Note that the certificates are only valid for 3 months, but you can set it up to auto-renew.

3.12 Running the network server:

3.13 Using systemd:

A systemd script for running the network server is available in the ‘scripts’ directory. Name of the script is:hil_network.service

3.14 Centos:

Centos uses systemd to controll all its processes.

Place the file hil_network.service under: /usr/lib/systemd/system/

3.15 Ubuntu:

Systemd is available from Ubuntu 15.04 onwards and LTS version 16.04 will ship with systemd by default.

Place the file hil_network.service under: /lib/systemd/system/

3.16 Starting the service:

Following commands will start the daemon: $ systemctl daemon-reload $ systemctl starthil_network

You can check the status using: $ systemctl status hil_network

To auto-start the service on boot: $ systemctl enable hil_network

3.17 For systems that do not support systemd:

Some systems like the LTS version of Ubuntu, Ubuntu 14.04 does not come with systemd pre-installed. It uses“Upstart” an equivalent of systemd to manage its daemons/processes.

For such systems, the networking server may be started as the HIL user by running:

$ hil-admin serve-networks &

To make this happen on boot, add the following to /etc/rc.local:


https://letsencrypt.org/

https://certbot.eff.org/


($ cd /var/lib/hil && su hil -c 'hil-admin serve-networks') &

3.18 HIL Client:

If your authentication backend is null, you only need to have the HIL_ENDPOINT defined in the client_env.In productions system where non-null backend is active, end users will have to include a username and password asadditional parameters in client_env file to be able to communicate with the hil server. If you created a admin userfor hil as a part of Setting Up HIL Database step, you will have to pass those credentials to HIL to be able to access,change state of HIL. Create a file client_env with following entries:

export HIL_ENDPOINT=http://127.0.0.1/export HIL_USERNAME=<hil_admin_username>export HIL_PASSWORD=<hil_admin_password>

To get started with HIL from your home dir do the following:

$ source client_env$ hil node list all

If you get an empty list [] as output then congratulations !! At this point, you should have a functional HIL servicerunning!

Tab completion is supported only for Bash. In order to activate that, copy the file scripts/hil-complete.sh to somelocation and put this in your bashrc:

. /path/to/hil-complete.sh

For more information see the [click documentation](http://click.pocoo.org/5/bashcomplete/)

3.19 Describe datacenter resources

For HIL to do anything useful, you must use the HIL API to populate the database with information about the resourcesin your datacenter – chiefly nodes, their NICs and the ports to which those NICs are attached. These are the relevantAPI calls:

• node_register

• node_delete

• node_register_nic

• node_delete_nic

• port_register

• port_delete

• port_connect_nic

• port_detach_nic

3.18. HIL Client: 15

http://click.pocoo.org/5/bashcomplete/



CHAPTER 4

INSTALL-devel-tldr

Following is the simplest way to get started with hacking HIL on Centos system. Assuming that

– You have a github account. – You have already forked the HIL repo. – You would be running HIL in apython virtual environment with SQLite DB as backend.

Install Dependencies:

yum install epel-release bridge-utils gcc httpd ipmitool libvirt \libxml2-devel libxslt-devel mod_wsgi net-tools python-pip python-psycopg2 \python-virtinst python-virtualenv qemu-kvm telnet vconfig virt-install

Clone repo:

git clone https://github.com/**username**/hilcd hil

Setting python virtual environment:

virtualenv .venvsource .venv/bin/activatepip install -e .[tests]

Configure HIL:

cp examples/hil.cfg.dev-no-hardware hil.cfg

Initialize database:

hil-admin db create

Start server:

hil-admin run-dev-server

From a separate terminal window:

17


cd ~/hil/virtualenv .venvsource .venv/bin/activatepip install -e .

Testing the setup:

hil node list all

If the above command reports an empty list. HIL is successfully installed and ready for hacking.

18 Chapter 4. INSTALL-devel-tldr

CHAPTER 5

INSTALL-devel

Following guide covers setting up a developement environment for HIL on a CentOS/RHEL based system.

5.1 Dependencies:

There are a few things that HIL expects the operating system to have:

yum install epel-release bridge-utils gcc httpd ipmitool libvirt \libxml2-devel libxslt-devel mod_wsgi net-tools python-pip python-psycopg2 \python-virtinst python-virtualenv qemu-kvm telnet vconfig virt-install

HIL requires a database server and currently supports only SQLite and PostgreSQL. If you choose to use PostgreSQLdatabase it is recommended to create a new system user with a separate home directory. This user will be configuredto control the hil database. The development environment will be created in its home directory.

5.2 Setting PostgreSQL for development environment:

If you choose to use sqlite database, skip this section and go to Getting Started with HIL Installation.

For setting up PostgreSQL, you will have to install the requisite packages on your system. Make sure your accountcan sudo to execute the following commands.

5.2.1 Part 1: Install PostgreSQL server.

Initialize the system. Configure PostgreSQL to allow password authentication.

Install the requisite packages on your server:

sudo yum install postgresql-server postgresql-contrib -y

Initialize postgresql:

19


sudo postgresql-setup initdb

Replace the term ident from following lines in file /var/lib/pgsql/data/pg_hba.conf with md5.

Before:

# "krb5", "ident", "peer", "pam", "ldap", "radius" or "cert". Note thathost all all 127.0.0.1/32 identhost all all ::1/128 ident#host replication postgres 127.0.0.1/32 ident#host replication postgres ::1/128 ident

After:

# "krb5", "ident", "peer", "pam", "ldap", "radius" or "cert". Note thathost all all 127.0.0.1/32 md5host all all ::1/128 md5#host replication postgres 127.0.0.1/32 ident#host replication postgres ::1/128 ident

Start postgresql service:

$ sudo systemctl start postgresql$ sudo systemctl enable postgresql

5.2.2 Part 2: Create a system user, database and database role.

Setting up development environment with PostgreSQL backend becomes easy with a dedicated user controlling thedatabase as well as the development environment.

Let that username be hil_dev. For simplicity we will use the same name for database and database role.

Create a new user on your system:

useradd --system -m -d /home/hil_dev -s /bin/bash -c "for HIL development" hil_dev

This will create a hil_dev user with following attributes:

hil_dev:x:1002:1002:for HIL development:/home/hil_dev:/bin/bash

exact uid/gid may vary depending on your system.

Switch to the hil_dev user:

sudo -u hil_dev -i

5.2.3 Setup database and role to control it.

Create database named hil_dev owned by user also named as hil_dev.

1. Create a database role named hil_dev with privileges to:

-r create roles-d create databases and-P will prompt for the password of the new user.

20 Chapter 5. INSTALL-devel


This is necessary since we have configured PostgreSQL to use password authentication:

sudo -i -u postgres$ createuser -r -d -P hil_devEnter password for new role: <Input password for database role hil>Enter it again: <Retype password for role hil>

Confirm that the role with requisite privileges is created as postgres user:

$ psql -c '\dg'List of roles

Role name | Attributes | Member of-----------+------------------------------------------------+-----------hil_dev | Create role, Create DB | {}postgres | Superuser, Create role, Create DB, Replication | {}

If you wish to delete the user. do the following as postgres user:

dropuser hil_dev

Note: Make sure that the database role you create corresponds to an existing system user. eg. There has to be a systemuser hil to access database named hil as database role named hil.

Create database hil_dev owned by database role hil_dev:

$ sudo -i -u hil_dev$ createdb hil_dev

Confirm it created a database named hil_dev and it is owned by hil_dev:

$ psql -c '\l'List of databases

Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+-----------------------hil_dev | hil_dev | UTF8 | en_US.UTF-8 | en_US.UTF-8 |postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +

| | | | | postgres=CTc/postgrestemplate1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +

| | | | | postgres=CTc/postgres

switch to user hil_dev. All subsequent installation steps assumes you are in the home directory of hil_dev

5.3 Getting Started with HIL Installation

First you will need to fork and clone the HIL repo into your dev VM.:

git clone https://github.com/**username**/hilcd hil

The first time you start working in the repository, set up a clean test environment:

virtualenv .venv

Enter the environment (do this every time you start working with HIL dev environment):

5.3. Getting Started with HIL Installation 21


source .venv/bin/activate

Then, proceed with installing the HIL and its dependencies into the virtual environment:

pip install -e .[tests]

The [tests] part pulls in dependencies only needed for running the test suite. There are several other “extras” youcan specify, which pull in dependencies needed for optional HIL features:

• postgres to use a PostgreSQL database.

• keystone-auth-backend for the keystone auth backend.

• keystone-client for keystone support in the client library and command line tool.

5.4 For older systems:

On systems with older versions of pip, such as Debian Wheezy and Ubuntu 12.04, this install will fail with thefollowing error:

AttributeError: 'NoneType' object has no attribute 'skip_requirements_regex'

Fix this by upgrading pip within the virtual environment:

pip install --upgrade pip

Versions of python prior to 2.7 don’t have importlib as part of their standard library, but it is possible to install itseparately. If you’re using python 2.6 (which is what is available on CentOS 6, for example), you may need to run:

pip install importlib

5.5 Setting up the Database:

The default dev environment uses SQLite as a database, so if you’re using it you can skip this section.

If you wish to use PostgreSQL instead, you may get an error psycopg2 package not found, when you dohil-admin db create in the next step. To avoid that problem, you may need to install some packages based onyour system type:

CentOS:

sudo yum install postgresql-devel

Ubuntu:

sudo apt-get install libpq-dev

After these packages have been installed, you’ll then need the python postgres driver in your HIL virtualenv:

pip install -e .[postgres]



5.6 Configuring HIL

Now the hil executable should be in your path. First, create a configuration file hil.cfg because if it’s notfound then hil would refuse to run and exit. There are two examples for you to work from, examples/hil.cfg.dev-no-hardware, which is oriented towards development, and examples/hil.cfgwhich is more productionoriented. These config files are well commented; read them carefully.

HIL can be configured using hil.cfg to not perform state-changing operations on nodes, headnodes and networks,allowing developers to run and test parts of a hil server without requiring physical hardware. To suppress actual nodeand headnode operations, set dry_run = True in the [devel] section.

5.6.1 If using PostgreSQL as a database backend

If you choose to use PostgreSQL and did the necessary steps as described above, put following string in hil.cfg undersection [database]:

uri = postgresql://hil_dev:<clear text password>@localhost:5432/hil_dev

It follows the format: postgresql://<user>:<password>@<address>/<dbname> where <user> is the name of thepostgres user you created, <password> is its password, <dbname> is the name of the database you created, and<address> is the address which hil should use to connect to postgres (In a typical default postgres setup, the rightvalue is localhost).

5.7 Setting up extensions

Most customizations require including extension names within the [extensions] section.

For suppressing actual network switch operations, use the mock switch driver :: hil.ext.switches.mock =

You can choose to disable authentication mechanism by enabling the null auth driver

hil.ext.auth.null =

5.8 Database auth

To enable an authentication mechanism, an appropriate authentication backend will need to be selected and enabled.Note that auth backends are mutually exclusive. For the database method (which stores users/passwords in the DB),add

hil.ext.auth.database =

Next initialize the database with the required tables:

hil-admin db create

5.9 Start the server

Run the server with the port number as defined in hil.cfg:

5.6. Configuring HIL 23


hil-admin run-dev-server --port <port no>

and in a separate window terminal:

hil-admin serve-networks

Finally, hil help lists the various API commands one can use. Here is an example session, testingheadnode_delete_hnic:

hil project_create projhil headnode_create hn proj img1hil headnode_create_hnic hn hn-eth0hil headnode_delete_hnic hn hn-eth0

5.10 Testing

Additionally, before each commit, run the automated test suite with py.test tests/unit. If at all possible, runthe deployment tests as well (py.test tests/deployment), but this requires access to a specialized setup, soif the patch is sufficiently unintrusive it may be acceptable to skip this step.

Testing contains more information about testing HIL. Migrations dicsusses working with database migrations andschema changes.


testing.html

migrations.html

CHAPTER 6

Upgrading HIL

This file describes the procedure for upgrading HIL to a new version.

1. Read the release notes for the particular release, below, which will cover any version-specific information.

2. Stop the HIL services (webserver and HIL network daemon). For exmaple:

$ systemctl stop httpd$ systemctl stop hil_network

3. Download and install the new version of HIL:

$ git clone https://github.com/cci-moc/hil$ python setup.py install

3. Upgrade the database:

$ hil-admin db upgrade heads

heads indicates that HIL core and all extensions should be upgraded together. This is the only workflow wesupport, but the curious can read the (developer-oriented) alembic documentation for more information:

• https://alembic.readthedocs.org/en/latest/

• https://alembic.readthedocs.org/en/latest/branches.html

4. If additional extensions have been added to hil.cfg, re-run hil-admin db create, which will createany tables needed by those extensions.

Note that removing extensions is not currently supported.

5. Restart the HIL services. e.g.:

$ systemctl start httpd$ systemctl start hil_network

25

https://alembic.readthedocs.org/en/latest/

https://alembic.readthedocs.org/en/latest/branches.html


6.1 Release Notes

6.1.1 0.5 (Upcoming)

• HIL now checks the permissions of hil.cfg on startup; if they are too permissive (i.e. non-owners may accessthe file) it will refuse to run.

• The CLI formats the output to be more readable. Output can be formatted as json by passing –json flag to anycommand.

6.1.2 0.4

• HIL now depends on OBMd to manage obms; the built in driver support has been removed. Upgrading requiresseveral steps: 1. Set up OBMd: https://github.com/CCI-MOC/obmd/ 2. Before updating HIL (i.e. while stillrunning 0.3):

– run hil-admin migrate-ipmi-info, supplying the OBMd base url and admin token. Seehil-admin migrate-ipmi-info --help. This will register all of your existing nodes withOBMd.

– Ensure that both the extensions hil.ext.obm.ipmi and hil.ext.obm.mock are listed in hil.cfg.

– Run hil-admin db create.

3. Update the HIL software as usual. Before restarting the serivces, remove both of the above obm extensionsfrom hil.cfg.

• New APIs to get a node’s power status and to power on a node.

• Fix a bug which prevented the deletion of a nic.

6.1.3 0.3

• Validation checks for the configuration file have been added; be advised that if there were errors in your hil.cfg that were previously undetected you may need to fix them before the servers will start again

• The interface to the hil command line tool has changed substantially; If you have custom scripts that invoke itthey will likely need to be modified. Use hil --help to explore the new interface.

• URLs for API calls are now prefixed with a version number, which for the current (unstable) API is v0; youmay need to update scripts and settings accordingly (although the HIL tools themselves should remain internallyconsistent).

Other changes, which do not require specific action when upgrading:

• All HIL APIs are now wrapped by the client library.

• Support for a new optional “maintenance pool” feature; see docs/maintenance-pool.md.

• Some new APIs (see docs/rest_api.md for details): * Networking actions can now be queried to get theirstatus. * When using the database auth backend, it is possible to list users.

• Updated APIs: * List networks now shows public networks to regular users. * Some switches now supportpublic key authentication. See

docs/network-drivers.md for details.

26 Chapter 6. Upgrading HIL

https://github.com/CCI-MOC/obmd/


6.1.4 0.2

HaaS was renamed to HIL in this release. Accordingly, there are several changes that need to be made when upgrading:

• Since the systemd service file is created manually, the old one needs to manually be deleted and the new onecopied in. * If wanting to keep the previous haas user, then the service file needs to

be modified to reflect the different username.

• The /var/lib/haas and /etc/haas.cfg entries need to be moved (or at least symlinked)

• You should remove the “haas” version, since it will be a different set of scripts: pip uninstall haas

• Re-copy hil.wsgi and update apache’s wsgi.conf entry to point to it.

• Update any scripts that have env vars (like HAAS_ENDPOINT`) to their HIL_ varieties.

6.1. Release Notes 27


28 Chapter 6. Upgrading HIL

CHAPTER 7

Using HIL as a Client

Interaction with HIL occurs via its REST API. The high-level semantics of the API are documented in API Description,and the mapping to HTTP is described in REST API.

The hil command line tool is a wrapper around this API. Running hil help will display an overview of theavailable commands. To tell hil which HIL instance to use, be sure to do one of:

1. Set the HIL_ENDPOINT environmental variable. An example (using the default port used when runninghil-admin run-dev-server) would be http://127.0.0.1:5000

2. Ensure that there is a hil.cfg in the current directory which contains a valid client section. A valid configfile in this case could look like

[client]endpoint = http://127.0.0.1:5000

• If both configuration methods are present, the HIL_ENDPOINT environmental variable will take precedenceover whatever is contained within hil.cfg.

• Though insignificant in some circumstances, the presence or absence of trailing slashes within the endpoint URLcan cause issues in communicating with the HIL server, such as “404” errors. For example, using http://127.0.0.1:5000 vs http://127.0.0.1:5000/.

If using the basic auth/database auth backend, you must set the environment variables HIL_USERNAME andHIL_PASSWORD to the correct credentials.

If using the auth/keystone auth backend, first make sure that the keystonemiddleware library is installed byrunning pip install keystonemiddleware. Next, ensure that there are OS environment variablesset for the following OpenStack authentication credentials: OS_AUTH_URL, OS_USERNAME, OS_PASSWORD,OS_PROJECT_NAME.

7.1 Deploying Machines

The most basic workflow for deploying machines onto a set of nodes allocated with HIL is as follows. First, create aheadnode, and attach it to the network that the hardware nodes PXE boot off of. Then, enter the headnode by VNCing

29

apidesc.html

rest_api.html


to it from the headnode host. The VNC port can be found with the REST show_headnode call. Authenticationsupport is slated for a future release. From there, you can set up SSH access to the headnode, or you can continue touse VNC if you prefer.

Next, configure DHCP and TFTP servers that will boot nodes into some automated install image. We have an exampleof this in examples/puppet_headnode. In this example, we use Kickstart to automate a Centos install. Ourkickstart file configures very little of the system, but complicated configuration can be done this way, especially byusing Kickstart to install a tool such as Puppet.

Our setup has one additional trick. We run a server that, firstly, serves the Kickstart file, but secondly makes it so eachnode only PXE boots the installer once. The last thing each node does while installing is to tell the server to delete itssymlink from the TFTP config, which will make the machine fall back to hard-disk booting the installed system.

This is, as the filepath states, merely an example of how you might deploy to physical nodes. Existing deploymentsystems such as Canonical’s MAAS have also been run succesfully.

7.2 Usage examples

Included here with are some examples about

• Interacting with HIL API directly using the curl utility.

• And using equivalent cli calls are also included.

7.2.1 1) Register a switch with HIL:

• Switch name: mockswitch01

• Switch type: mock

• Host name: switchhost01

• User name: switchuser01

• Password: password1234

api call

curl -X put http://127.0.0.1:5000/v0/switch/mockswitch01 -d '{"type": "http://schema.massopencloud.org/haas/v0/switches/mock","hostname": "switchhost01","username": "switchuser01","password": "password1234"}'

cli call

$ hil switch register mockswitch02 mock switchhost01 switchuser01 password1234

7.2.2 2) Registering a Node

• Node name: dummyNoderHIL-02

• OBMd uri: ‘‘https://obmd.example.com/node/node-02‘‘

• OBMd admin token: d7c9e587124fb1cb0d03f79933a9ce28

For a node with the above information, use the following api call:

30 Chapter 7. Using HIL as a Client



curl -X PUT http://127.0.0.1:5001/v0/node/dummyNode01 -d '{> "obmd": {> "uri": "https://obmd.example.com/node/node-02",> "admin_token": "d7c9e587124fb1cb0d03f79933a9ce28"> }> }'

Corresponding cli calls will be as follows:

$ hil node register dummyNoderHIL-02 \https://obmd.example.com/node/node-02 \d7c9e587124fb1cb0d03f79933a9ce28

7.2.3 3) Creating a Project in HIL

• Project name: mockproject01

api call

curl -X put http://127.0.0.1:5000/v0/project/mockproject01

cli call

$ hil project create mockproject01

7.2.4 4) Register a node in HIL

• Node name: mocknode01

• Node type: mock

• Host name: mockhost01

• User name: nodeuser01

• Password: password1234

api call

curl -X put http://127.0.0.1:5000/v0/node/mocknode01 -d '> {"type": "http://schema.massopencloud.org/haas/v0/nodes/mock",> "hostname": "mockhost01",> "username": "nodeuser01",> "password": "password1234"}'

cli call

$ hil node register mocknode01 mock mockhost01 nodeuser01 password1234

7.2.5 4) Creating a Network in HIL

• Network name: mocknetwork01

• owner: admin

• access: mockproject

7.2. Usage examples 31


• net_id: 101

api call

curl -X put http://127.0.0.1:5000/v0/network/mocknetwork01 -d '{"owner": "admin","access": "mockproject","net_id": "101"}'

cli call

$ hil network create mocnetwork01 admin mockproject 101

32 Chapter 7. Using HIL as a Client

CHAPTER 8

Extensions

HIL supports a simple extension mechanism, to allow external plugins to implement things we don’t want in the HILcore. One obvious example of this is drivers.

Extensions are python modules. The extensions section in hil.cfg specifies a list of modules to import onstartup, for example:

[extensions]hil.ext.driver.switch.dell =hil.ext.driver.switch.complex_vlan =hil.ext.driver.obm.ipmi =some_3rd_party.hil.drivers.obm.robotic_power_button_pusher

If the extension requires any kind of initialization, it may define a function setup, which will be executed afterall extensions have been loaded. This function must accept arbitrary arguments (for forwards compatibility), but atpresent must not rely on the presence or value of any argument.

If the extension needs its own configuration options, it may read them from a section with the same name as themodule, e.g.:

[extensions] some_3rd_party.hil.drivers.obm.robotic_power_button_pusher

[some_3rd_party.hil.drivers.obm.robotic_power_button_pusher] push_duration = 3 seconds

Extensions should not make use any part of the HIL source tree that does not explicitly invite it (i.e. everything bydefault is Private). Components which may be used from extensions will explicitly say so in their documentation,(and describe in detail how they may be used). Extension-approved components currently include:

• Most of hil.network_allocator

• hil.auth

• From hil.model:

– db.Model

– Switch

• The migration framework; see Migrations for an overview.

33

migrations.html


See the docstrings for each component for details.

Additionally, extensions may add wsgi middleware to the flask application from their setup function. For example:

app.wsgi_app = my_middleware(app.wsgi_app)

Note that the order in which the setup functions are run is not defined. As such, if multiple extensions add wsgimiddleware the order in which they are applied is also undefined. Using more than one such extension is discouraged.An ordering may be defined in the future.

34 Chapter 8. Extensions

CHAPTER 9

Overview

This file describes the HIL API. We first describe the main objects, then users and security model, and finally providea full reference for the API.

By default, HIL is using the unstable “v0” version for all its API.

9.1 Objects in the HIL

• project - a grouping of resources (e.g., headnodes, nodes, networks).

• node - a physical node. Either unallocated or belongs to a project. Has one or more NICs attached to it.

• headnode - a controlling machine for a project, today a VM, assigned to one project

• NIC - network card, identified by a user-specified label (e.g., PXE, ipmi, user1, silly) will have a visible eth-ernet mac address (or equivalent unique number for other network types), and is always part of one node andconnected to at most one port.

• HNIC - headnode network card, identified by a user-specified label (e.g., PXE, ipmi, user1, silly), and is alwayspart of one headnode.

• port - a port to which NICs can be connected. Only visible to admins.

• network - a network, today implemented as a VLAN, that NICs and HNICs can be connected to. See net-works.md for more details.

The authentication system is pluggable. Authentication mechanisms are provided by extensions (see extensions.md), but the rules about who is allowed to access what are dictated by HIL core. Operations etiher require administratiepriilieges, or access to a particular project.

Of note, “users” are not a concept that HIL core understands, though some of the individual auth extensions do.

35


9.2 API design philosophy

We provide the most basic API that we can, and attempt to impose no structure that is not required for authorizationpurposes.

• A ‘project’ is merely an authorization domain. It is reasonable to have logically independent groupings ofresources within one project, but the HIL will not help you create such a structure. Policies like this belong inhigher-level tools built on top of the hil.

• We considered having a mechanism for staging a large number of networking changes and performing themall-together, and even potentially allowing roll-back. Instead, we simply have API calls to connect a NIC to anetwork, and to disconnect it. All other functionalities can be built on top of this.

There is no garbage-collection of objects. If an object is being used somehow, it cannot be deleted. For example, if anode is on a network, the user can neither de-allocate the node nor delete the network. They must first detach the nodefrom the network. (The one exception to this is that, when deleting a headnode, all of its HNICs are deleted with it.This is due to a technical limitation—we cannot currently dynamically add and remove HNICs.)

Most objects are identified by “labels” that are globally unique, e.g., nodes, networks, groups, users, headnodes. Whilewe may eventually change this, it seems a reasonable limitation for now that simplifies the implementation and willallow potential sharing of resources. The system will return an error if a second user tries to create an object with analready existing label. The one exception is NICs, where the label is unique only on a per-node basis.

36 Chapter 9. Overview

CHAPTER 10

API Reference

10.1 How to read

Each possible API call has an entry below containing:

• an HTTP method and URL path, including possible <parameters> in the path to be treated as arguments.

• Optionally, a summary of the request body (which will always be a JSON object).

• A human readable description of the semantics of the call

• A summary of the response body for a successful request. Many calls do not return any data, in which case thisis omitted.

• Any authorization requirements, which could include:

– Administrative access

– Access to a particular project or

– No special access In general, administrative access is sufficient to perform any action.

• A list of possible errors.

In addition to the error codes listed for each API call, HIL may return:

• 400 if something is wrong with the request (e.g. malformed request body)

• 401 if the user does not have permission to execute the supplied request.

• 404 if the api call references an object that does not exist (obviously, this is acceptable for calls that create theresource).

Below is an example.

10.1.1 my_api_call

POST /url/path/to/<thing>

37


Request Body:

{"some_field": "a value","this-is-an-example": true,"some-optional-field": { (Optional)

"more-fields": 12352356,...

}}

Attempt to do something mysterious to <thing> which must be a coffee pot, and must not be in use by other users.If successful, the response will include some cryptic information.

Response Body (on success):

{"some-info": "Hello, World!","numbers": [1,2,3]

}

Authorization requirements:

• No special access.

Possible errors:

• 418, if <thing> is a teapot.

• 409, if:

– <thing> does not exist

– <thing> is busy

• {"foo": <bar>, "baz": <quux>} denotes a JSON object (in the body of the request).

10.2 Core API Specification

API calls provided by the HIL core. These are present in all installations.

10.2.1 Networks

network_create

PUT /network/<network>

Request Body:

{"owner": <owner>,"access": <access>,"net_id": <net_id>

}

Create a network. For the semantics of each of the fields, see Networks.


38 Chapter 10. API Reference

./networks.html


• If net_id is '' and owner and access are the same project, then access to that project is required.

• Otherwise, administrative access is required.

Possible errors:

• 409, if a network by that name already exists.

• See also bug #461

network_delete

DELETE /network/<network>

Delete a network. The user’s project must be the owner of the network, and the network must not be connected to anynodes or headnodes. Finally, there may not be any pending actions involving the network.


• If the owner is a project, access to that project is required.

• Otherwise, administrative access is required.

Possible Errors:

• 409 if:

– The network is connected to a node or headnode.

– There are pending actions involving the network.

show_network

GET /network/<network>

View detailed information about <network>.

The result must contain the following fields:

• “name”, the name of the network

• “channels”, description of legal channel identifiers for this network. This is a list of channel identifiers, withpossible wildcards. The format of these is driver specific, see below.

• “owner”, the name of the project which created the network, or “admin”, if it was created by an administrator.

• “access”, a list of projects that have access to the network or null if the network is public

• “connected-nodes”: nodes and list of nics connected to network

Response body (on success):

{"name": <network>,"channels": <chanel-id-list>,"owner": <project or "admin">,"access": <project(s) with access to the network/null>"connected-nodes": {"<node>": [<list of nics connected to network]}

}


• If the network is public, no special access is required.

10.2. Core API Specification 39


• Otherwise, access to a project in the “access” list or administrative access is required.

• Admins and network owners can see all nodes connected to network; other users only see connected nodes thatthey have access to.

Channel Formats

802.1q VLAN based drivers

Channel identifiers for the VLAN based drivers are one of:

• vlan/native, to attach the network as the native (untagged) VLAN

• vlan/<vlan_id> where <vlan_id> is a VLAN id number. This attaches the network in tagged, mode,with the given VLAN id.

Additionally, the show_networks api call may return the channel identifier vlan/*, which indicates that anyVLAN-based channel id may be used.

Where documentation specifies that the network driver should choose a default channel, the VLAN drivers choosevlan/native.

list_networks

GET /networks

List all networks.

Returns a JSON dictionary of dictionaries, where the exterior dictionary is indexed by the network name and the valueof each key is another dictionary with keys corresponding to that network’s id and projects

Response contains all networks if the user is an admin. Otherwise, response only contains all public networks.

The response must contain the following fields:

• “network”, the name of a network

• “network_id”, the id of the network

• “projects”, a list of projects with access to the network or ‘None’ if network is public

Example Response for an admin user:

{"netA": {

"network_id": "101","projects": ["qproj-01", qproj-02"]

},"netB": {

"network_id": "102","projects": None

}}

Example Response for a regular user:

{"netA": {

"network_id": "102",





"projects": None},"netB": {

"network_id": "103","projects": None

}}


• Administrative access is required to list all networks

• No special access is required to list all public networks

list_network_attachments

GET /network/<network>/attachments

List all nodes that are attached to network .

If optional argument ‘project’ is supplied, only attached nodes belonging to the specified project will be listed.

Returns a JSON dictionary of dictionaries with first level key being the name of the attached node and second levelkeys being:

• “nic”, the name of the nic on which the node is attached

• “channel”, the channel on which the attachment exists

• “project”, the name of the project which owns the attached node

Example Response:

{"node1": {

"nic": "nic1","channel" "vlan/native","project": "projectA"

},"node2": {

"nic": "nic2","channel": "vlan/235","project": "projectB"

}}


• Admins or the project that is the owner can list all attached nodes.

• Other projects can only list their own nodes.

network_grant_project_access

PUT /network/<network>/access/<project>

Add to access list for .




• Only admins or the network owner can grant a project access to the network.

Possible Errors:

• 404 - If the network or project does not exist

• 409 - If the project already has access to the network

network_revoke_project_access

DELETE /network/<network>/access/<project>

Remove from access list for .


• Only admins, the network owner, or the project itself can revoke a project’s access to the network.

Possible Errors:

• 404 - If the network or project does not exist.

• 409 - If the project is the owner of the network.

node_connect_network

POST /node/<node>/nic/<nic>/connect_network

Request body:

{"network": <network>,"channel": <channel> (Optional)

}

Connect the network named <network> to <nic> on <channel>.

<channel> should be a legal channel identifier specified by the output of show_network, above. If <channel>is omitted, the driver will choose a default, which should be some form of “untagged.”

Networks are connected and detached asynchronously. If successful, this API call returns a status code of 202 Ac-cepted, and queues the network operation to be performed. Each nic may have no more than one pending networkoperation; an attempt to queue a second action will result in an error.

It is important that users of this API check the status of their request using the show_networking_action APIto ensure that their previous calls were successful before queuing any new actions on the same nic.

Response body:

{"status_id": <unique_id>,

}


• Access to the project to which <node> is assigned.

• Either <network>must be public, or its "access" field must name the project to which <node> is assigned.

Possible errors:

• 409, if:



– The current project does not control <node>.

– The current project does not have access to <network>.

– There is already a pending network operation on <nic>.

– <network> is already attached to <nic> (possibly on a different channel).

– The channel identifier is not legal for this network.

node_detach_network

POST /node/<node>/nic/<nic>/detach_network

Request body:

{"network": <network>

}

Detach <network> from <nic>.

Networks are connected and detached asynchronously. If successful, this API call returns a status code of 202 Ac-cepted, and queues the network operation to be preformed. Each nic may have no more than one pending networkoperation; an attempt to queue a second action will result in an error.

Just like the node_attach_network API, please check the status using the show_networking_action APIto check the status of previous calls before queuing any new actions.

Response body:


}


• Access to the project to which <node> is assigned.

Possible Errors:

• 409, if:

– The current project does not control <node>.

– There is already a pending network operation on <nic>.

– <network> is not attached to <nic>.

10.2.2 Nodes

node_register

PUT /node/<node>

Register a node with optional metadata

The "obmd" field has two subfields. The first is "uri", which is the uri for the given node under obmd’s API; ifobmd’s API is available at https://obmd.example.com, then node4 will have a uri of https://obmd.example.com/node/node4. The second is "admin_token", which is the admin token for obmd.

Note that different nodes may be managed by different instances of obmd; HIL does not care.



Request Body:

{"obmd": {

"uri": "https://obmd.example.com/node/node4","admin_token": "208eb61c77c3468558ccd9eec9ca0598"

},"metadata": { (Optional)

"label_1": "value_1","label_2": "value_2"

}}


• Administrative access.

Possible errors:

• 409, if a node with the name <node> already exists

node_delete

DELETE /node/<node>

Delete the node named <node> from the database.



node_register_nic

PUT /node/<node>/nic/<nic>

Request Body:

{"macaddr": <mac_addr>

}

Register a nic named <nic> belonging to <node>. <mac_addr> should be the nic’s mac address. This isn’t usedby HIL itself, but is useful for users trying to configure their nodes.



Possible errors:

• 409 if <node> already has a nic named <nic>

node_delete_nic

DELETE /node/<node>/nic/<nic>

Delete the nic named <nic> and belonging to <node>.





node_enable/disable_obm

PUT /node/<node>/obm

Request body:

{"enabled": <boolean>

}

Enable or disable management of the node’s OBM. The OBM must be disabled to detach the node from a project, andmust be enabled to do things like power cycle the node, view the console etc.

Accepts a boolean argument indicating whether to enable or disable the obm.

Enabling an obm which is already enabled, or disabling one that is already disabled will silently no-op.


• Access to the project to which <node> is assigned (if any) or administrative access.

node_power_cycle

POST /node/<node>/power_cycle

Request Body:

{"force": <boolean> (Optional, defaults to False)

}

Power cycle the node named <node>, and set it’s next boot device to PXE. If the node is powered off, this turns it on.

Accepts one optional boolean argument that determines whether to soft (default) or hard reboot the system.



Possible Errors:

• 409, if the node’s OBM is not enabled (see node_enable_obm).

node_set_bootdev

PUT /node/<node>/boot_device

Request body:

{"bootdev": <boot device>

}

The request body consists of JSON with a bootdev argument:

Sets the node’s next boot device persistently



Possible Errors:




For IPMI devices

The valid/allowed boot devices are:

• pxe : do a pxe boot (network boot)

• disk: boot from local hard disk

• none: to reset boot order to default.

node_power_off

POST /node/<node>/power_off

Power off the node named <node>. If the node is already powered off, this will have no effect.



Possible Errors:


node_power_on

POST /node/<node>/power_on

Power on the node named <node>. If the node is already powered on, this will have no effect.



Possible Errors:


node_power_status

GET /node/<node>/power_status

Returns the node’s power status.

Response body:

["power_status": "<IPMI chassis power status response>"

]

Response examples: “on” or “off” for IPMI, or “Mock Status” for mock OBM.



Possible Errors:




show_console

GET /node/<node>/<console

Stream output from the node’s serial console.

The response body will be a byte stream from the node’s serial console; streaming will continue until the connectionis broken.

Only one client may be connected to a node’s console in this fashion at a time; if a second client connects, the firstwill be disconnected.

The node’s obm must be enabled; a BlockedError will be raised otherwise.



Possible Errors:


list_nodes

GET /nodes/<is_free>

Return a list of all nodes or free/available nodes. The value of is_free can be all to return all nodes or free toreturn free/available nodes.

Response body:

["node-1","node-2",...

]


• No special access

list_project_nodes

GET /project/<project>/nodes

List all nodes belonging to the given project

Response body:

["node-1","node-2",...

]


• Access to <project> or administrative access



list_project_networks

GET /project/<project>/networks

List all networks belonging to the given project

Response body:

["network-1","network-2",...

]



show_node

GET /node/<node>

Show details of a node.

Returns a JSON object representing a node. The object will have at least the following fields:

• “name”, the name/label of the node (string).

• “project”, the name of the project a node belongs to or null if the node does not belong to a project

• “nics”, a list of nics, each represented by a JSON object having at least the following fields:

– “label”, the nic’s label.

– “macaddr”, the nic’s mac address.

– “networks”, a JSON object describing what networks are attached to the nic. The keys are channels andthe values are the names of networks attached to those channels.

– “port”, the port to which the nic is connected to or null if the nic is not connected to any port. This field isonly visibile if the caller is an admin.

– “switch”, the switch that has the port to which the nic is connected to or null if the nic is not connected toany port. Just like port, this is only visible if the caller is an admin.

• “metadata”, a dictionary of metadata objects

Response body when run by a non-admin user:

{"metadata": {

"EK": "pk"},"name": "node1","nics": [

{"label": "nic1","macaddr": "01:23:45:67:89","networks": {

"vlan/235": "storage","vlan/native": "pxe"

}





},{

"label": "nic2","macaddr": "12:34:56:78:90","networks": {}

}],"project": "project1"

}

Response body when run by an admin:

{"metadata": {

"EK": "pk"},"name": "node1","nics": [

{"label": "nic1","macaddr": "01:23:45:67:89","networks": {

"vlan/235": "storage","vlan/native": "pxe"

},"port": "gi1/0/1","switch": "dell-01"

},{

"label": "nic2","macaddr": "12:34:56:78:90","networks": {},"port": null,"switch": null

}],"project": "project1"

}


• If the node is free, no special access is required.

• Otherwise, access to the project to which <node> is assigned is required.

• Admin acces to view port and switch information.

10.2.3 Projects

project_create

PUT /project/<project>

Create a project named <project>





Possible Errors:

• 409, if the project already exists

project_delete

DELETE /project/<project>

Delete the project named <project>



Possible Errors:

• 409, if:

– The project does not exist

– The project still has resources allocated to it:

* nodes

* networks

* headnodes

project_connect_node

POST /project/<project>/connect_node

Request body:

{"node": <node>

}

Reserve the node named <node> for use by <project>. The node must be free.


• Access to <project> or administrative access.

Possible errors:

• 404, if the node or project does not exist.

• 409, if the node is not free.

project_detach_node

POST /project/<project>/detach_node

{"node": <node>

}



Return <node> to the free pool. <node> must belong to the project <project>. It must not be attached to anynetworks, or have any pending network actions.

If a maintenance pool is configured, the maintenance service will be notified and the node will be moved into themaintenance project. Otherwise, the node will go directly to the free pool. Read docs/maintenance-pool.mdfor more information.



Possible errors

• 409, if the node is attached to any networks, or has pending network actions, or if the node’s obm is enabled.

list_projects

GET /projects

Return a list of all projects in HIL

Response body:

["manhattan","runway",...

]



node_set_metadata

PUT /node/<node>/metadata/<label>

Request Body:

{"value": <value>

}

Set metadata with <label> and <value> on <node>.



node_delete_metadata

DELETE /node/<node>/metadata/<label>

Delete metadata with <label> on <node>.





10.2.4 Headnodes

headnode_create

PUT /headnode/<headnode>

Request body:

{"project": <project>,"base_img": <base_img>

}

Create a headnode owned by project <project>, cloned from base image <base_img>. <base_img> must beone of the installed base images.



Possible errors:

• 409, if a headnode named <headnode> already exists

headnode_delete

DELETE /headnode/<headnode>

Delete the headnode named <headnode>.


• Access to the project which owns <headnode> or administrative access.

headnode_start

POST /headnode/<headnode>/start

Start (power on) the headnode. Note that once a headnode has been started, it cannot be modified (adding/removinghnics, changing networks), only deleted — even if it is stopped.



headnode_stop

POST /headnode/<headnode>/stop

Stop (power off) the headnode. This does a force power off; the VM is not given the opportunity to shut down cleanly.





headnode_create_hnic

PUT /headnode/<headnode>/hnic/<hnic>

Create an hnic named <hnic> belonging to <headnode>. The headnode must not have previously been started.



Possible errors:

• 409, if:

– The headnode already has an hnic by the given name.

– The headnode has already been started.

headnode_delete_hnic

DELETE /headnode/<headnode>/hnic/<hnic>

Delete the hnic named <hnic> and belonging to <headnode>. The headnode must not have previously beenstarted.



Possible errors:

• 409, if the headnode has already been started.

headnode_connect_network

POST /headnode/<headnode>/hnic/<hnic>/connect_network

Request body:

{"network": <network>

}

Connect the network named <network> to <hnic>.

<network> must be the name of a network which:

1. the headnode’s project has the right to attach to, and

2. was not assigned a specific network id by an administrator (i.e. the network id was allocated dynamically byHIL). This constraint is due to an implementation limitation, but will likely be lifted in the future; see issue#333.

Additionally, the headnode must not have previously been started.

Note that, unlike nodes, headnodes may only be attached via the native/default channel (which is implicit, and maynot be specified).

Rationale: separating headnodes from hil core is planned, and it has been deemed not worth the development effort toadjust this prior to the separation. Additionally, headnodes may have an arbitrary number of nics, and so being able toattach two networks to the same nic is not as important.





• Either <network> must be public, or its "access" field must name the project which owns <headnode>.

Possible errors:


headnode_detach_network

POST /headnode/<headnode>/hnic/<hnic>/detach_network

Detach the network attached to <hnic>. The headnode must not have previously been started.



Possible errors:


list_project_headnodes

GET /project/<project>/headnodes

Get a list of names of headnodes belonging to <project>.

Response body:

["<headnode1_name>","<headnode2_name>",...

]



show_headnode

GET /headnode/<headnode>

Get information about a headnode. Includes the following fields:

• “name”, the name/label of the headnode (string).

• “project”, the project to which the headnode belongs.

• “hnics”, a JSON array of hnic names that are attached to this headnode.

• “vncport”, the vnc port that the headnode VM is listening on; this value can be null if the VM is powered offor has not been created yet.

• “uuid”, UUID for the headnode.

• “base_img”, the os image that the headnode is running.

Response body:



{"name": <headnode>,"project": <projectname>,"nics": [<nic1>, <nic2>, ...],"vncport": <port number>,"uuid": <headnode uuid>,"base_img": <headnode base_img>

}



10.2.5 Switches

10.2.6 show_switch

GET /switch/<switch>

View detailed information about <switch>.

The result must contain the following fields:

• “name”, the name of the switch

• “ports”, a list of the name of the ports which exist on the switch

Response body (on success):

{"name": <switch>,"ports": <ports-list>

}



switch_register

Register a network switch of type <type>

<type> (a string) is the type of network switch. The possible values depend on what drivers HIL is configuredto use. The remainder of the fields are driver-specific; see the documentation for the driver in question (in docs/network-drivers.md.

PUT /switch/<switch>

Request body:

{"type": <type>,(extra args; depends on <type>)

}





Possible Errors:

• 409, if a switch named <switch> already exists.

switch_delete

DELETE /switch/<switch>

Delete the switch named <switch>.

Prior to deleting a switch, all of the switch’s ports must first be deleted.



Possible Errors:

• 409, if not all of the switch’s ports have been deleted.

list_switches

GET /switches

Return a list of all switches registered in HIL

Response body:

["switch1","hickory",...

]



switch_register_port

PUT /switch/<switch>/port/<port>

Register a port <port> on <switch>.

The permissible values of <port>, and their meanings, are switch specific; see the documentation for the appropriatedriver for more information.



Possible Errors:

• 409, if the port already exists



switch_delete_port

DELETE /switch/<switch>/port/<port>

Delete the named <port> on <switch>.

Prior to deleting a port, any nic attached to it must be removed.



Possible Errors:

• 409, if there is a nic attached to the port.

port_connect_nic

POST /switch/<switch>/port/<port>/connect_nic

Request body:

{"node": <node>,"nic": <nic>

}

Connect a port a node’s nic.



Possible errors:

• 404, if no port is connected to the given nic.

• 409, if the nic or port is already attached to something.

port_detach_nic

POST /switch/<switch>/port/<port>/detach_nic

Detach the nic attached to <port>.



Possible errors:

• 404, if the port is not attached to a nic

• 409, if the port is attached to a node which is not free.

port_revert

POST /switch/<switch>/port/<port>/revert

Detach the port from all networks. This is an asynchronous call. It’s a good idea to check the status using theshow_networking_action API before queuing any new actions on the same port.



Response body:


}



Possible errors:

• 404, if there is no nic attached to port

• 409, if there is already a networking action pending on port

show_port

GET /switch/<switch>/port/<port>

Show the node and nic to which the port is connected.

Response body:

{"node": "mynode","nic": "mynic","networks": {"vlan/1511": "mynetwork"}

}

If there is no nic attached to a port, the response body is just an empty json object:

{}



Possible errors:

• 404, if the switch and/or port do not exist.

list_active_extensions

GET /active_extensions

Response Body:

[ “hil.ext.switches.mock”, “hil.ext.network_allocators.null”, . . . ]

List all active extensions.



10.3 API Extensions

API calls provided by specific extensions. They may not exist in all configurations.



10.3.1 The hil.ext.auth.database auth backend

list_users

GET /auth/basic/users

List all users

Response body:

{'hil_user': {'is_admin': True, 'projects': ["runway"]}'mock_user': {'is_admin': False, 'projects': ["manhattan"]}

}



user_create

PUT /auth/basic/user/<username>

Request body:

{"password": <plaintext-password>"is-admin": <boolean> (Optional, defaults to False)

}

Create a new user whose password is <plaintext-password>.



Possible errors:

• 409, if the user already exists

user_delete

DELETE /auth/basic/user/<username>

Delete the user whose username is <username>



user_set_admin

‘PATCH /auth/basic/user/’

Request Body:

{ ‘is_admin’: }

Set admin status of user ‘’ to true (admin user) or false (regular user)

10.3. API Extensions 59




Possible errors:

• 404, if the user does not exist.

• 409, if the user tries to set own admin privilege

user_add_project

POST /auth/basic/user/<user>/add_project

Request Body:

{ “project”: <project_name> }

Add a user to a project.



user_remove_project

POST /auth/basic/user/<user>/remove_project

Request Body:

{ “project”: <project_name> }

Remove a user from a project.



show_networking_action

GET /networking_action/<status_id>

Get the status of the networking call queued by node_connect_network, node_detach_network, or port_revert, where<status_id> is returned by any of the network calls.

Response Body:

{ “status”: , “node”: <node-label>, “nic”: <nic-label>, “new_network”: <network-name> “type”: “channel”: }

where:

• status can either be “DONE”, “PENDING”, or “ERROR”.

• new_network can be null in case of node_detach_network or revert_port.

• type can be revert_port or modify_port.

• channel could be ‘’ in case of revert_port.

The status of a networking call is kept until a new action on the same nic is added, after which the old entry is deleted.




• Access to the project which owns the node that has the nic on which the networking action is active, or admin-istrative access.

Possible errors:

• 404, if the status_id is not found.

10.3. API Extensions 61



CHAPTER 11

Deployment

This is a guide to the best practices for deploying Hardware Isolation Layer (HIL). It documents many of the assump-tions that the HIL development team made in terms of how HIL would be used, how it would be deployed, and theconfiguration of the hardware systems it would be managing.

For instructions on installing HIL, take a look at Installation. For a general overview, please take a look at Introduction.

11.1 Nodes

11.1.1 BIOS/Firmware

We recommend coming up with a standard list of BIOS settings for your nodes and documenting them. These couldvary between pools of different types, like virtualization based (which could benefit more from hyperthreading) vsHPC (which might not).

Settings useful to document would include:

• BIOS boot order

• Hyperthreading

• Whether turbo boosting is enabled

Boot order

A provider should consistently set up all nodes to have the same order. It is recommended that this order consist of:cdrom, disk, PXE. By placing CDROM first, the datacenter administrator can easily boot the system locally in orderto perform maintenance tasks like upgrading the BIOS. Having disk come before PXE allows customers who installan OS to have it boot first, reducing boot times and avoiding accidental overwriting of their installed OS.

• Note: In the future, we anticipate a feature (see issue 303) where customers can explicitly request PXE bootingfor the next boot via an API call, however the permanent boot order would not likely be able to be changed asthat is not supported by IPMI in a standard way.

63

INSTALL.html

README.html



Devices

For higher security, it may be desirable to disable CD/DVD access to the OS, only enabling it when needed formaintenance or other reasons. This is because an inadvertent CD/DVD left in the drive could allow a user to burn animage. Placed in another system or left in the tray, this image could then allow an attack on another system.

Firmware security

End users will have full control your node. If they are not trusted, it is extremely important to disable firmware updatemechanisms before access is first given to it. The less reachable this is through remote access (ie - hardware jumpersvs. software settings) the more secure the system should be.

Ideally, hardware that does not trust the user to update firmware should be used. Examples of this are AMD Seamicrosystems as well as HP Moonshot. Short of that, it is recommended that one lock down the firmware options as muchas possible in case access were ever gained to the boot screens or IPMI interface. This includes setting a BIOS menupassword as well as disabling any kind of privileged access from within the OS.

Firmware updates for the motherboard and for peripheral devices should be disabled via hardware (ideally) or softwaremechanisms, as they could represent a vector by which a user (whether maliciously or unintentionally) could impactfuture users of a machine and possibly violate their security. It is important to note that administrators should alsomonitor the manufacturers’ websites for legitimate firmware upgrades and should regularly update nodes themselves,as these could contain security or other fixes. Here is an incomplete list of firmwares to which these guidelines apply:

• BIOS Should be configured not to allow updates from within the OS, since even the root user is untrusted.

• Hard disks

• Network cards

• GPUs

• Other peripherals

11.2 Networks

11.2.1 Switch configuration

• In order to deploy HIL, at least one switch supported by one of HIL’s drivers is required. See the list of supportedswitches

• HIL uses the ssh command to connect to the switches. As a consequence, administrators will need to connectto the switch once via ssh, as the user HIL will run as. This is because otherwise ssh will display the usualconfirmation dialog for new hosts:

Are you sure you want to continue connecting (yes/no)?

. . . which the HIL drivers do not handle.

• Please make sure that there are no unwanted VLANs on a switchport when it is registered in HIL. There aremore details about this in the description for supported switches.

11.2.2 VLANs

The network administrator will need to pre-allocate a set of VLANs to dedicate to HIL’s management. These are keptwithin the vlan section of the config file.

64 Chapter 11. Deployment




11.2.3 Headnodes

Currently, the system or systems that run the HIL service (sometimes informally called the HIL master) also run theHeadnode virtual machines. These can be used to access and administrate user networks, whether public or private.To facilitate this, all VLANs that are allocatable by HIL must be trunked using 802.1Q to HIL service systems. Thisport upon which these networks are shared is indicated using the trunk_nic member of the [headnode] section.

11.2.4 Multiple Switches

HIL supports networks spanning multiple switches via 2 mechanisms:

1. Vendor-specific “stacking” features that make all of the switches appear to be one big one.

2. By running a network cable between the two switches, and setting all VLANs belonging to HIL to be trunked(tagged) on the interfaces connecting those switches.

IMPORTANT: If you’re operating a multi-switch deployment using technique (2) above, you should be sure to con-figure the switches to prevent VLAN hopping attacks::

https://en.wikipedia.org/wiki/VLAN_hopping

11.2. Networks 65


66 Chapter 11. Deployment

CHAPTER 12

Consistency model

Here is the consistency model for headnodes.

• headnode_create: After running this, you can then run headnode_create_nic,headnode_delete_nic, headnode_connect_network, headnode_detach_network asmuch as you want, until you run headnode_start. The headnode’s VM is then created, started, andconnected to the appropriate networks. As soon as you do this, the headnode becomes ‘frozen’, and no morechanges to it are allowed. (Currently, the headnode is marked dirty/clean instead of unfrozen/frozen. This linesup with the semantics in one way, in that a dirty headnode hasn’t been fully applied yet. But, they act differentenough that this will probably change. This change will not affect external behavior.)

• headnode_delete: This deletes the headnode immediately, detaching it from all networks it was attachedto. Due to current limitations, this operation cannot be run at all. Eventually, this call should succeed as long asthe headnode is powered off.

• headnode_start, headnode_stop: These cycle power on the headnode. It’s also possible that, eventu-ally, we might allow networking changes to powered-off headnodes. (It’s semantically reasonable, but might betricky in implementation.)

The following steps will let you install, configure PostgreSQL server and create a hil database for development orproduction version of HIL on a Centos - 7 server.

For simplicity of configuration and ease of maintenance we will use single name hil for creating a system user,database role and database name. This is merely a guideline for users new to database setup and administration.Experienced users are free to choose any other method that suits their need. The end goal is to have a way to have aworking PostgreSQL backend for hil

67


68 Chapter 12. Consistency model

CHAPTER 13

Part 1: Install PostgreSQL server.

Initialize the system. Configure PostgreSQL to allow password authentication.

1. Install the requisite packages on your server.

sudo yum install postgresql-server postgresql-contrib -y

2. Initialize postgresql:

sudo postgresql-setup initdb

3. Replace the term ident from following lines in file /var/lib/pgsql/data/pg_hba.conf with md5

Before:

# "krb5", "ident", "peer", "pam", "ldap", "radius" or "cert". Note thathost all all 127.0.0.1/32 identhost all all ::1/128 ident#host replication postgres 127.0.0.1/32 ident#host replication postgres ::1/128 ident

After:

# "krb5", "ident", "peer", "pam", "ldap", "radius" or "cert". Note thathost all all 127.0.0.1/32 md5host all all ::1/128 md5#host replication postgres 127.0.0.1/32 ident#host replication postgres ::1/128 ident

4. Start postgresql service

$ sudo systemctl start postgresql$ sudo systemctl enable postgresql

##Part 2: Setup system user and database role for HIL.

Create database named hil owned by user also named as hil.

69


5. Create a system user hil:

If you are setting up this database for a production setup then you might have already created a system user formanaging HIL. In that case skip this step.

If you are setting up database for development purpose you will need to create a system user to manage HIL databaseas follows.

format: useradd <username> --system -d <path-to-home-dir> -m -r

In this case both system username and home directory will be named hil

useradd hil --system -d /var/lib/hil -m -r

6. Create a database role named hilwith privileges to: -r create roles -d create databases and -Pwill prompt forthe password of the new user. This is necessary since we have configured PostgreSQL to use password authentication.

$ sudo -u postgres createuser -r -d -P hilEnter password for new role: <Input password for database role hil>Enter it again: <Retype password for role hil>

Confirm that the role with requisite privileges is created as postgres user:

$ psql -c '\dg'List of roles

Role name | Attributes | Member of-----------+------------------------------------------------+-----------hil | Create role, Create DB | {}postgres | Superuser, Create role, Create DB, Replication | {}

Note: It is recommended that the PostgreSQL role and database you create correspond to an existing system user.eg. There should be a system user hil to access database named hil as database role named hil. Advanceduser/role/database configurations may not need to follow this rule. More information is available in the DatabaseRoles and Privileges reference guide.

7. Create database hil owned by database role hil:

sudo -u hil createdb hil

confirm it created a database named hil and it is owned by hil.

$ psql -c '\l'List of databases

Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+-----------------------hil | hil | UTF8 | en_US.UTF-8 | en_US.UTF-8 |postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +

| | | | | postgres=CTc/postgrestemplate1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +

| | | | | postgres=CTc/postgres

##Finally:

If you have followed all steps so far. Put following string in hil.cfg under section [database]

uri = postgresql://hil:<clear text password >@localhost:5432/hil

It follows the format: postgresql://<user>:<password>@<address>/<dbname> where <user> is thename of the postgres user you created, <password> is its password, <dbname> is the name of the database you

70 Chapter 13. Part 1: Install PostgreSQL server.

INSTALL.html

INSTALL-devel.html

https://www.postgresql.org/docs/9.0/static/user-manag.html

https://www.postgresql.org/docs/9.0/static/user-manag.html


created, and <address> is the address which hil should use to connect to postgres (In a typical default postgressetup, the right value is localhost).

Continue with installation steps:

continue with production install or continue with development install

71

INSTALL.html

INSTALL-devel.html


72 Chapter 13. Part 1: Install PostgreSQL server.

CHAPTER 14

Keystone Authentication

An authentication backend for Openstack’s Keystone is maintained in this source tree as hil.ext.auth.keystone. This document describes its configuration and usage in detail.

NOTE: The HIL command line interface only supports the Keystone v3 API. The server supports anything supportedby keystonemiddleware.

14.1 Usage

Once HIL has been configured to work with Keystone, an administrator must manually add Openstack projects to HILbefore they can access the HIL API. The HIL project names must correspond to the Openstack UUIDs. For example,an administrator may execute the command:

hil project_create 00de7c85e594473db7461cdf7367166a

To grant the Openstack project with that UUID access to HIL.

Note that the plugin recognizes any user with an admin role on any project as a HIL administrator, similar to thedefault policy for core Openstack projects. This is true even for projects not that do not exist within HIL; such projectswill not be able to own resources (such as nodes networks, etc), but may perform admin-only operations (such ascreating projects).

The HIL command line interface will look for the same OS_* environment variables used by the Openstack commandline tools; these may be set by a user to authenticate when using the CLI.

A script to set these variables correctly can be downloaded from the Openstack web dashboard via “Access & Security.”

14.2 Configuration

As with any other extension, you must load the extension in hil.cfg:

73

http://docs.openstack.org/developer/keystonemiddleware/


[extensions]hil.ext.auth.keystone =

The backend must then be configured to talk to your Keystone server. The Keystone project maintains documentationon how to do this at:

http://docs.openstack.org/developer/keystonemiddleware/middlewarearchitecture.html

Configuring HIL to talk to Keystone deviates in the following ways:

• The paste configuration is not used; you can simply ignore the sections that refer to paste.

• The options that the Keystone documentation puts in the section [keystone_authtoken] should insteadbe placed in the extension’s section in hil.cfg, i.e. [hil.ext.auth.keystone].

14.3 Debugging Tips

If authentication is not working with HIL, first check if authentication to OpenStack is working. Using the Open-Stack CLI, run the command: openstack token issue -f value -c id. If a text token is returned, thenauthentication to OpenStack is working.

Testing authentication directly to the HIL API is also helpful. Using the token from the tip above, run: curl -H'x-auth-token: <token>' <HIL address>/nodes/free. If the response lists the nodes in the currentHIL setup, then the Keystone middleware has been setup correctly.

74 Chapter 14. Keystone Authentication

http://docs.openstack.org/developer/keystonemiddleware/middlewarearchitecture.html

CHAPTER 15

Logging

15.1 Overview

HIL supports basic logging of the API calls received in the below format, recording the user who issued the API call(or guest if the user was unauthenticated) and the time. Log files are rotated daily.

2016-03-31 14:55:07,961 - hil.rest - INFO - (guest) - API call: list_projects()

15.2 Setup

Logging can be configured with the following options in hil.cfg.

[general]log_level = DEBUGlog_dir = /var/log/hil

log_dir specifies the directory where the log files will be stored. The HIL user must have write permissions to thisdirectory. If the option is omitted logging to files is disabled.

log_level specifies the logging level to record. Valid options are: CRITICAL, DEBUG, ERROR, FATAL, INFO,WARN, WARNING. The default value is WARNING, but an option of INFO is recommended for an API log (A log levelof INFO is set for API calls).

For more information on logging visit the python 2 documentation.

This document describes how to work with HIL’s database migration framework. It is aimed at developers; for in-structions on upgrading a production database see UPGRADING.rst.

75

https://docs.python.org/2/howto/logging.html#when-to-use-logging


76 Chapter 15. Logging

CHAPTER 16

Introduction

HIL uses SQLAlchemy as a database abstraction layer, Flask-SQLAlchemy for integration with Flask, and Flask-Migrate/Alembic for migration support. This document explains HIL-specific details; refer to the libraries’ documen-tation for general information about these tools. This document mainly focuses on Alembic and Flask-Migrate.

HIL uses migration scripts for updating the database to work with a new version of the software. Any time thedatabase schema changes, a migration script must be written so that existing installations may be upgraded to the nextHIL release automatically.

16.1 Background

Developers unfamiliar with alembic should at least skim the Alembic tutorial, which explains general alembic con-cepts. Developers should also read the overview documentation for Flask-Migrate, which integrates Alembic withFlask.

16.2 Usage In HIL

HIL uses the migration libraries as follows:

The Flask-Script extension is used to expose Flask-Migrate’s functionality, via the hil-admin db subcommand,implemented in hil/commands/. This command is almost exactly as described in the Flask-Migrate documenta-tion, except we’ve added sub-command hil db create, which initially creates and populates the database tables.This command is idempotent, so it is safe to run it on an already-initialized database. When adding extensions tohil.cfg, the create command should be re-run to create any new tables; see UPGRADING for details.

We use Alembic’s branch support to handle tables created by extensions; Developers should be familiar with this partof alembic’s interface. Each extension should have its own head. We do not merge these heads.

When a new migration script needs to be written, the procedure depends on whether or not the migration scriptconcerns an extension which does not already have any migration scripts.

77

http://www.sqlalchemy.org/

http://flask-sqlalchemy.pocoo.org/2.1/

http://flask.pocoo.org/

https://flask-migrate.readthedocs.org/en/latest/


http://alembic.readthedocs.org/en/latest/

http://alembic.zzzcomputing.com/en/latest/tutorial.html


http://flask-script.readthedocs.org/en/latest/


http://alembic.readthedocs.org/en/latest/branches.html


16.2.1 If the extension does not have migration scripts already

First, choose a directory in which to store the migration scripts for the extension. Storing the migrations as close to themodule as possible is recommended. Add code to your extension to register this directory with hil.migrations.paths, e.g.:

from hil.migrations import paths:

paths[__name__] = 'Path to my migrations directory'

Make sure the value will be correct regardless of where the source tree is located; making it a function of __file__is a good idea. Have a look at existing extensions for examples if need be. hil.ext.switches.dell is a goodexample.

Also, make sure that the migration scripts will be included in the final package. This usually just means adding anappropriate entry to package_data in setup.py. If the extension is maintained outside of the HIL core sourcetree, make sure you pass zip_safe=False to setup.

Next, to generate a template migration, execute:

hil-admin db migrate --head hil@head \--branch-label ${name_of_extension_module} \-m '<Summary of change>'

Alembic will look at the differences between the schema in the database and the one derived from the HIL source, andgenerate its best attempt at a migration script. The script will be stored in hil/migrations/versions. Move itto the directory you chose, and change the value of down_revision to None.

Sanity check the output; Alembic often does a good job generating scripts, but it should not be trusted blindly.

Finally, run hil-admin db upgrade heads to upgrade the database according to your new migration.

16.2.2 If the extension already has migrations, or the script is for HIL core

To generate a new migration script, execute:

hil-admin db migrate --head ${branch_name}@head \-m '<Summary of change>'

Where ${branch_name} is either the name of the extension’s module (if the script is for an extension) or hil (ifthe script is for HIL core).

Alembic will look at the differences between the schema in the database and the one derived from the HIL source, andgenerate its best attempt at a migration script. The script will be stored in hil/migrations/versions.

The value of down_revision should be the identifier of the previous migration script. The value ofbranch_labels should be ('<branch_name>',) where branch_name should match what was used in thecommand to generate the migration script. Finally, the value of branch_lables in the previous migration script(named by down_revision in the new one) must be set to None. This will need to be changed manually.

Sanity check the output; Alembic often does a good job generating scripts, but it should not be trusted blindly.

78 Chapter 16. Introduction


16.3 Notes on Generating Migrations and Checking the Output

16.3.1 State of the Database

For automatic migrations the database being loaded to generate the migration should match the schema of the currentmaster branch. To ensure this, create a new PostgreSQL database and initialise it using hil-admin db createwhile on a branch that is up to date with current HIL master branch. The command to generate the migration scriptshould then be run after checking out the branch that has the changes that the script should be generated for.

16.3.2 Renaming Columns

Alembic will not rename a column, instead it will delete the original column and create a new one withthe new name. This is an issue as the data will then not be contained in the new column (see Data Mi-gration vs Schema Migration below). To change the name of a column the script should be edited man-ually to remove the lines dropping the old column and creating one with the new name and replace themwith a line altering the column: op.alter_column(u'<table_name>', '<old_column_name>',new_column_name='<new_column_name>')

16.3.3 Data Migration vs. Schema Migration

The migrations Alembic generates are schema migrations: they will create/delete tables, columns, and relationships,but they do not populate these with data. This can be an issue, particularly in the case of a new relationship that wouldapply to existing data. None of the existing data will be accessible via the new relationship unless the data itself isspecifically migrated as well.

This can be done by directly encoding data within the script and using a command like bulk_insert(), executinga SQL statement to SELECT the data and INSERT it into the new column, or by creating a live interaction with thedatabase.

lines 28-32 in hil/migrations/versions/89630e3872ec_network_acl.py show an example of exe-cuting a SQL statement then using bulk_insert() to migrate data.

16.4 Writing tests

The file tests/unit/migrations.py provides some basic infrastructure for testing migrations. The commentsthere describe things in full detail, but the basic strategy is, back up a database with a known set of objects usingpg_dump, and place the result in tests/unit/migrations/. To generate a dump, execute::

pg_dump -U $hil_user $database_name --column-inserts > filename.sql

--column-inserts is necessary to ensure that the result can be loaded via SQLAlchemy; the default is not validSQL and takes advantage of specific features of the psql command. You will also need to edit the file manually,doing the following:

• It is likely that the system you’re developing on will have a newer version of postgres than what travis uses, soyou may need to remove statements setting options that don’t exist int he older versions. Delete any statementsof the form SET option_name = ..., where option_name is any of:

– idle_in_transaction_session_timeout

– lock_timeout

– row_security

16.3. Notes on Generating Migrations and Checking the Output 79


• Delete all statements using the keywords GRANT, REVOKE, or EXTENSION. These will cause permissionerrors if your database user is not root.

• Delete all statements of the form ALTER TABLE ... OWNER TO ...; these may cause failures if theconnection details on the machine which runs the tests are different from yours, since the refer to specificdatabase users/roles.

• At the top of the file, document the following:

– In roughly what stage of HIL’s development this dump was taken, e.g. “Just after re-integrating flask” or“first integration of openvpn”

– The git commit hash of the revision of HIL which created the database.

– A list of extensions that were loaded into HIL when the database was created.

– Any other options in the config file that would have affected the contents of the database.

– How the database was generated, e.g. “by the function hil.foo.bar as of commit $hash.”

80 Chapter 16. Introduction

CHAPTER 17

Network Drivers

This document describes the network driver model, primarily from a system administrator’s standpoint, though thereare some details useful for developers writing tools which interact with the API as well.

17.1 Overview

There are two classes of network-related drivers: network allocators and switches. Switches are straightforward tounderstand; they provide support for a particular hardware switch. Network allocators are a bit more abstract.

HIL networks may be backed by various underlying isolation technologies (currently the only network allocatorshipped with HIL uses 802.1q VLANs, but there may be more, e.g. VXLANs in the future). A network alloca-tor manages the details of mapping networks created by HIL to these underlying technologies. For example, thevlan_pool allocator maps each network to a unique VLAN id.

Drivers are implemented as extensions, and must be added to the [extensions] section of hil.cfg. You mustsupply exactly one network allocator driver, and one or more switches. For example::

...[extensions]hil.ext.network_allocators.vlan_pool =hil.ext.switches.dell =hil.ext.switches.nexus =

Some drivers may also need driver specific options, which go in a section with the same name as the extension, e.g.::

[hil.ext.network_allocators.vlan_pool]vlans = 300-500, 700-750

17.2 Network allocator drivers

The only network allocator shipped with HIL that is of interest to users (there are others useful for developmentpurposes) is the VLAN allocator. The name of the extension is hil.ext.network_allocators.vlan_pool,

81


and it requires a single extension-specific config option, vlans, which is a comma separated list of VLAN ids and/orranges of VLAN ids. Networks created within HIL will use VLANs specified in the configuration file. An example::

...

[extensions]hil.ext.network_allocators.vlan_pool =...

[hil.ext.network_allocators.vlan_pool]vlans = 300, 500-700, 800-950...

Once HIL has been started, removing VLANs from this list is not supported. You may add additional VLANs, but youwill have to re-run hil-admin db create.

17.3 Security

It is VERY IMPORTANT that you be sure to configure your switches to guard against VLAN hopping attacks:

https://en.wikipedia.org/wiki/VLAN_hopping

Doing so is not difficult, and it is critical for security.

17.4 Switch drivers

At present, all switch drivers shipped with HIL require that the VLAN pool allocator is in use. There are five switchdrivers shipped with HIL:

• hil.ext.switches.dell, which provides a driver for the Dell Powerconnect 5500 series switches.

• hil.ext.switches.nexus, which provides a driver for some Cisco Nexus switches. Only the 3500 and5500 have been tested, though it is possible that other models will work as well.

• hil.ext.switches.brocade, for the brocade VDX 6740.

• hil.ext.switches.n3000, for Dell N3000 series switches.

• hil.ext.switches.dellnos9, for Dell switches running Dell Networking OS 9.

Different switches may or may not have certain capabilities. The show_switch call can be used to see what a switchis capable of. Currently, HIL exposes the following switch capabilities:

• nativeless-trunk-mode : If supported, a switchport can be configured to have no native networks intrunk mode. If not supported, then the switchport must be first connected to a native network before adding anytagged VLANs.

• There should be no “enable” password for switch users which will be used by HIL.

Per the information in rest_api.md, the details of certain API calls are driver-dependant, below are the details foreach of these switches.

82 Chapter 17. Network Drivers


17.4.1 Powerconnect 5500 driver

Switch preparation

A few commands are necessary to run on the switch before it can be used with HIL.

1. Every VLAN that could be used on the switch must first be enabled on the switch itself. This is distinct fromadding a VLAN to a port. To enable all VLANs to work with the switch, run this command (note that it seemsto time out or something, but it just takes a while):

configure vlan 2-4094

2. This switch uses ssh for connection. Be sure that ssh is enabled on the switch.

3. If you choose to login using the public key, then provide any string as the password. The user running the HILnetwork daemon should have access to the private key.

4. For any switchport that you register, please make sure that there are no VLANs added to it in trunk mode.

switch_register

To register a Dell Powerconnect switch, the "type" field of the request body must have a value of::

http://schema.massopencloud.org/haas/v0/switches/powerconnect55xx

In addition, it requires three extra fields: "username", "hostname", and "password", which provide thenecessary information to connect to the switch via telnet ("hostname" may also be an IP address). SSH supportis planned, but even so we do not recommend allowing connectivity to a switch’s management interface from anuntrusted network.


Port names must be of the same form accepted by the switch’s console interface, e.g. gi1/0/5. Be very careful whenspecifying these, as they are not validated by HIL (this will be fixed in future versions).

17.4.2 Dell N3000 driver

Switch preparation

1. Just like the Powerconnect 5500, every VLAN that could be used on the switch must first be enabled on theswitch. To enable all VLANs to work with the switch, run this command:

# configure# vlan 2-4093

Note: VLAN range for the Dell N3000 is shorter than the other switches’.

1. HIL uses ssh to connect to these switches. Configure the switch to accept ssh connections.


17.4. Switch drivers 83


switch_register

To register a Dell N3000 switch, the "type" field of the request body must have a value of::

http://schema.massopencloud.org/haas/v0/switches/delln3000

It requires the same fields as the powerconnect driver, plus an additional field “dummy_vlan” like the nexus driver,which should be a JSON number corresponding to an unused VLAN id on the switch. Unlike the nexus switch, thisdummy vlan must exist otherwise you cannot set it as the native vlan, but this vlan should not be used for any networks.


Register ports just like the powerconnect driver. e.g. gi1/0/5.

17.4.3 Nexus driver

Switch preparation

1. This switch uses ssh for connection. Be sure that ssh is enabled on the switch.


switch_register

The type field for the Nexus driver has the value::

http://schema.massopencloud.org/haas/v0/switches/nexus

The nexus driver requires the same additional fields as the powerconnect driver, plus an additional field “dummy_vlan”,which should be a JSON number corresponding to an unused VLAN id on the switch. This VLAN should be deacti-vated (and thus no traffic should flow across it ever). This exists to get around an implementation problem related todisabling the native VLAN.

For example, if you’ve chosen VLAN id 2222 to use as the dummy vlan, on the switch’s console, run:

# config terminal# no vlan 2222

The body of the api call request can then look like:

{"type": "http://schema.massopencloud.org/haas/v0/switches/nexus","username": "MyUser","password": "secret","hostname": "mynexus.example.com","dummy_vlan": 2222

}

• If you choose to login using the public key, then provide any string as the password. Also, the user running theHIL network daemon should have access to the private key.




Like the powerconnect driver, the Nexus driver accepts port names of the same format accepted by the underlyingswitch, in this case (e.g.) ethernet 1/42. The same concerns about validation apply.

17.4.4 Brocade driver

Switch preparation

1. Make sure the REST server is enabled on the switch.

sw0# configsw0(config)# rbridge-id 1sw0(config-rbridge-id-1)# no http server shutdown

1. For every switchport that is registered in HIL, make sure that there are no VLANs attached to that port. Togglingswitchport command usually does the trick. The switchport should be in access mode with VLAN 1.

sw0# configsw0(config)# interface TenGigabitEthernet 1/0/4sw0(conf-if-te-1/0/4)# no switchportsw0(conf-if-te-1/0/4)# switchportsw0(conf-if-te-1/0/4)# do show running-config interface TenGigabitEthernet 1/0/4

switch_register

The type field for the Brocade NOS driver has the value:

http://schema.massopencloud.org/haas/v0/switches/brocade

In addition to type, the brocade driver requires three additional fields hostname, username, password, andinterface_type. interface_type refers to the type and speed of the ports on the switch, ex. “TenGiga-bitEthernet”, “FortyGigabitEthernet”. If you have multiple types of ports on the same switch, register the switchmultiple times with different parameters for interface_type.

The body of the api call request will look like:

{"type": "http://schema.massopencloud.org/haas/v0/switches/brocade","username": "MyUser","password": "secret","hostname": "mybrocade.example.com","interface_type": "TenGigabitEthernet"

}


The brocade driver accepts interface names the same way they would be accepted in the console of the switch, ex.101/0/10.

17.4. Switch drivers 85


17.4.5 Dell Networking OS 9 driver

Switch preparation

1. Make sure the REST server is enabled on the switch.

Dell-S3048-ON#configMOC-Dell-S3048-ON(conf)#rest-server http

1. When a port is registered, ensure that it is turned off (otherwise it might be sitting on a default native vlan). HILwill then take care of turning on/off the port.

switch_register

The type field for the Dell Networking OS 9 driver has the value:

http://schema.massopencloud.org/haas/v0/switches/dellnos9

In addition to type, you need to supply username, password, hostname, and interface_type.interface_type is of the form “GigabitEthernet”, “TenGigabitEthernet”. Please look at the switch guide formore valid interface types.

The switch’s API server either runs on port 8008 (HTTP) or 8888 (HTTPS), so be sure to specify that in thehostname.

This switch must have a native VLAN connected first before having any trunked VLANs. The switchport is turned ononly when a native VLAN is connected.

If you have multiple types of ports on the same switch, register the switch multiple times with different parameters forinterface_type.

The body of the api call request will look like:

{"type": "http://schema.massopencloud.org/haas/v0/switches/dellnos9","username": "MyUser","password": "secret","hostname": "switch.example.com:8008","interface_type": "GigabitEthernet"

}


It accepts interface names the same way they would be accepted in the console of the switch, ex. 1/3.

17.4.6 Using multiple switches

Networks managed by HIL may span multiple switches. No special configuration of HIL itself is required; just registereach switch as normal and ensure that all VLANs in the allocator’s vlans option are trunked to every managed switch.



17.5 Openvswitch Driver (For development purpose only. Do not usein production.)

This driver is made available so that developers can have real switch like functionality without having access to anyreal switch hardware. Following documentation is relevant for Openvswitch 2.5.0 To get started you will need to:

1. install openvswitch in your development machine (or VM):

yum install openvswitch #For fedora# Following script will:# * enable the openvswitch service;# * Start the openvswitch server;# * Show the status to the user.

systemctl enable openvswitchsystemctl start openvswitchsystemctl status openvswitchovs-vsctl show

1. create a bridge and name it <switch_name>:

ovs-vsctl add-br <switch_name>

1. Add some ports to this bridge:

ovs-vsctl add-port <switch_name> <port_name>

These are just illustrative examples. You may have to do more to setup your switch before using it with HIL.

For setup on other linux flavors refer to the openvswitch documentation.

Optionally, a script is made available for reference at::

https://github.com/SahilTikale/HIL_contrib/blob/master/hilInYourLap/create_datacenter.→˓sh

Warning: Use the script at your own discretion.

To register the driver with HIL, you will need the openvswitch bridge name.

17.5.1 switch_register

To register an openvswitch, the "type" field of the request body must have a value of::

http://schema.massopencloud.org/haas/v0/switches/ovs

In addition, it requires one extra field: "ovs_bridge" which provides the bridge name already created in open-vswitch.

17.5.2 switch_register_port

Openvswitch accepts any string for port name. Once you add a port to the openvswitch, you can register the same withHIL.

Notice: Bridge and Ports must pre-exist in the openvswitch before registering them with HIL.

17.5. Openvswitch Driver (For development purpose only. Do not use in production.) 87


17.5.3 Using multiple switches

Use-cases that involve configurations requiring access to multiple switches can be achieved by adding bridges ofdifferent names to openvswitch and registering them as separate switches and ensure that all VLANs in the allocator’svlans option are trunked to every managed switch.


CHAPTER 18

Types of Networks in the HIL

Networks in the HIL have three fields:

• ‘owner’ represents what project owns it; or ‘admin’ if it was administrator-created. If there are still networksowned by a project, then that project cannot be deleted. Only the owner has the ability to delete a network.

• ‘access’ represents which projects’ nodes can access it, or [] (the empty list) if all projects’ nodes can.

• ‘allocated’ represents whether the underlying network ID was taken from the network driver’s allocation pool.(The other option is that it was manually specified by an administrator.) This is important for network deletion.

The command line tool’s network_create treats an empty string to mean that the network should be allo-cated; any other argument is treated as a network ID.

These fields are not really independent of each other. Here are the legal combinations, with some explanation:

• (admin, all, yes): Public network internal to HIL

• (admin, all, no): Public network that connects outside the HIL

• (admin, project, yes): External provisioning network for one project

• (admin, project, no): (kind of useless, but legal)

• (project, project, yes): Normal project-created network

Here are the illegal ones:

• (project, [anything], no): Normal users cannot just assert control of external VLANs.

• (project, all, yes): In the same vein, user-created public networks are also not allowed.

• (project, different project, yes): Projects can grant access to other networks later using ‘net-work_grant_project_access’, but must have the project that is the owner of the network on the access list.

89


90 Chapter 18. Types of Networks in the HIL

CHAPTER 19

Network Teardown

The script create_dell_vlans can be used to pre-populate the bridges and vlan nics needed for the HIL tooperate, but right now we don’t have an automated way to delete them. You shouldn’t need to do this to use the HIL,but if for some reason you want to delete the nics related to a given vlan, you can do:

brctl delif br-vlan${vlan_number} em3.${vlan_number}vconfig rem em3.${vlan_number}ifconfig br-vlan${vlan_number} downbrctl delbr br-vlan${vlan_number}

We’re using em3 as an example; do a brctl show to find out what the right nic actually is.

91


92 Chapter 19. Network Teardown

CHAPTER 20

Testing

20.1 Overview

Testing for HIL is done using pytest. All tests are placed in the “tests” subdirectory, which is subdivided into:

• unit - for basic unit tests. These are safe to run without a full HIL enviornment (i.e. you don’t need libvirtetc).

• deployment - for tests that need to run against an actual setup, with libvirtd and at least one real switch.

Developers should run at least the unit and pycodestyle tests before making a commit. Ideally, the deployment testsshould also be run, though we’re less strict about this. Developers should also introduce tests for any new func-tions/methods they write, or any new or fixed functionality. Ideally, bug fixes should include a test (whether unit orfunctional) that reproduces the problem scenario so that we can ensure the bug doesn’t return!

20.2 Configuration

Install HIL and its test dependencies in the virtual environment, done via: pip install -e .[tests] in theroot directory. This will pull a recent version of pytest and you’ll be ready to test the code. Using pip’s -e optioninstalls the hil in editable mode, which has the advantage that one need not reinstall every time a file is changed!

Most of the tests use a common set of default configuration options, as seen in testsuite.cfg.default. Ifyou wish to override these parameters, you may copy it to testsuite.cfg and edit. In the future, this will allowdevelopers to do things like test against different DBMSes (for the moment some changes are needed to the test suiteto actually support this). For now, it’s mostly interesting when running the deployment tests (see below).

20.3 Running

To run all tests, from the HIL root directory run:

93

http://pytest.org/


py.test tests/

To run just a subset of them, specify a particular file or directory:

py.test tests/unitpy.test tests/unit/api.py

To run just a pycodestyle test:

pycodestyle *.py tests/ hil/

As stated above, running at least tests/unit is mandatory before each commit.

20.4 Code Coverage

The pytest-cov plugin (included in requirements.txt) allows us to generate reports indicating what code is/is not ex-ecuted by our tests. setup.cfg configures pytest to output coverage information by default, so you mostly don’tneed to worry about this from a usage perspective – just make sure you pay attention to the output.

More information is available on the projects PyPI page.

20.5 Test structure

Tests are kept in the tests directory, which is further organized into a few subdirectories:

• unit

• integration

• deployment

For each file in the hil code, there should be a file with the same name in the unit directory. Within those files, classes(class names must begin with “Test”) can be used to organize tests into functional areas. Function names must alsobegin with “test”. See tests/unit/api.py for examples.

There also may be a few files loose in the tests directory that do not clearly fit into one of the above categories.

20.6 Integration tests

The tests/integration directory contains tests that require substantial setup of external software, or specialconfiguration. These tend to be expensive in terms of time. These are run automatically by our travis-ci configuration.

20.7 Deployment tests

The deployment tests (tests/deployment) are a set of unit tests which are most useful when executed in an envi-ronment with real hardware and a libvirtd instance available. Some of these are executed by our travis-ci configurationusing mock drivers, but (obviously) not against real hardware. They should be run before merging any patch relatingto specific hardware support, or interacting with headnodes. To run the deployment tests, you must do the following:

94 Chapter 20. Testing

https://pypi.python.org/pypi/pytest-cov


• Write a testsuite.cfg reflecting your environment. Copy examples/testsuite.cfg-deployment and edit. In particular, you will need to load the extensions for your switch driversand the corresponding network allocator (see drivers.md), and specify extension-specific options.

• Write a site-layout.json describing the layout of your environment. The file examples/site-layout.json provides an example. Here is a full description of the file format:

site-layout.json must contain a single json object, with two fields: "switches" and "nodes".

"switches" must be a list of json objects, each of which describes a switch in your environment, and must have thesame fields as required in the body of the switch_register API call (see rest_api.md), plus a "switch"field, which supplies the name of the switch (normally specified in the URL).

"nodes" must be a list of json objects, each of which defines a node, and has three fields:

• "name", a string which specifies the name of the node.

• "nics", a list of objects each describing a nic on the node, with the fields (all of them strings):

– "name", the name of the nic

– "mac", the mac address of the nic

– "switch", the name of the switch that the nic is connected to

– "port", the name/label of the port on the switch that the nic is connected to

• "obm", An object with the same set of fields as required by the obm field in the node_register API call.

The tests currently require at least four nodes to be specified in site-layout.json, each of which must have atleast one nic connected to the switch.

20.7. Deployment tests 95


96 Chapter 20. Testing

CHAPTER 21

VLAN Primer

This document is a brief primer on 802.1q VLANs (Virtual LANs): what they are, how switches tend to deal withthem, how Linux deals with them, and how the HIL uses them.

21.1 Managed vs. Unmanaged Switches

The primary function of an unmanaged switch is to connect several computers into a single network segment. This issimilar to the purpose of a hub - which is in essence just a shared wire - except that it understand the ethernet protocoland thus does not have to send traffic to every machine. Instead it can only send traffic to the machines which need tosee it. This improves performance somewhat, but for the most part does not affect usage of the device.

For a managed switch, the basic purpose is the same. However, a managed switch provides additional capabilities, andtypically can be configured via a web interface, serial console, or telnet or ssh.

One piece of additional functionality provided by nearly all managed switches is support for VLANs, which aretypically used to create logical networks within the switch - i.e. they allow the switch to be divided into isolatedsections, which cannot communicate with one another.

21.2 What are VLANs?

At a most basic level, a VLAN is simply a 12 bit tag in the header of an ethernet packet, which can be used bynetworking equipment. Mostly this is used as described above - it provides both the switch and other devices withinformation about which logical network the packet belongs to.

21.3 How do switches deal with VLANs?

Typically, traffic moving “through” a managed switch will have a VLAN tag associated with it, and will be distributedto all of the ports associated with that VLAN. A port may be in one of two modes: access or trunking. (Some switcheshave additional modes, but they are usually variations on one of these, and inessential).

97


Access mode is the simpler of the two. For a port in access mode:

• The port is associated with exactly one VLAN tag.

• Any device(s) connected to the port do not and need not have any knowledge of the use VLANs.

• Any traffic coming in on the port has the associated VLAN tag added.

• Any traffic going out on the port has it’s VLAN tag removed.

Access mode allows a port to be a member of a logical network without requiring any knowledge of VLANs on thepart of whatever is connected to the port.

In trunking mode:

• The port is associated with zero or more “tagged” VLANs.

• The port may also have one “native” VLAN.

• If an ethernet packet coming in on the port has no VLAN tag, then it is tagged with the “native” VLAN’s tag.

• If an ethernet packet coming in on the port has a VLAN tag, then:

– If that VLAN is associated with the port, it is left untouched, and distributed to the corresponding logicalnetwork as normal

– Otherwise, the packet is dropped.

• Packets bound for VLANs which are “tagged” for the port retain their VLAN tags when going out on th port.

• TODO: my intuition is that packets bound for the native VLAN have their tags stripped off before exiting theport, but I’m not 100% sure of this.

Trunking mode could actually fully replace the functionality of access mode, since a trunked port with just a nativeVLAN (and no tagged VLANs) is equivalent. It is more flexible however.

21.4 How does Linux deal with VLANs?

In a typical networking configuration, a Linux machine is not VLAN aware. However, it is possible to create VLANaware network interfaces with the vconfig command. If eth0 is a physical network interface, the command

vconfig add eth0 104

will create an interface called eth0.104. Any ethernet traffic coming into the machine on eth0 that is tagged forVLAN 104 will come out of eth0.104, with its VLAN tag stripped off. Any traffic going into eth0.104 will havea VLAN tag for 104 added, and then be sent out on eth0. eth0.104 may be deleted with:

vconfig rem `eth0.104`

21.5 How does the HIL use VLANs?

At present, when a network consisting of nodes 0, 1 and 2, and headnode H, using vlan tag N is deployed,

• The ports to which 0, 1 and 2 are attached are configured for trunked mode, and given access to VLAN N.Depending on the user’s preferences, the VLAN may either be tagged or native, on a per-port basis.

• An interface ethM.N is created on the HIL master, where ethM is a network interface connected to the switch.ethM must be connected to a port that is configured as trunking, with VLAN M being tagged for that port.

• ethM.N is attached to H (not directly, see(1)).

98 Chapter 21. VLAN Primer


• H is started.

From here, we have an isolated network connecting 0, 1, 2 and H. H can then manage the nodes by e.g. providing aPXE server and invoking the HIL API to reboot the nodes.

(1): The one slight wrinkle with the network setup described above is that libvirt doesn’t actually allow connectingethM.N directly to the VM; instead we connect both ethM.N and the virtualized network interface to a bridge, whichachieves the same effect.

21.5. How does the HIL use VLANs? 99


100 Chapter 21. VLAN Primer

CHAPTER 22

Driver Model

HIL supports different types of devices (like network switches, out of band management controllers) through a drivermodel. This document describes the general concepts of that driver model and where it is currently applied.

The HIL API manipulates each type of device (currently just network devices) through the same set of API calls,meaning that code within the main HIL tree need not worry about platform-specific details (like how to add a VLANto a Cisco 5500 switch port).

Most objects within HIL are persisted to the database, and thus classes map to tables. The driver makes use ofSQLAlchemy’s joined table inheritance to map class hierarchies to the database.

Each type of object that supports different drivers has a top-level superclass, e.g. Switch or OBM, and the driversthemselves are subclasses of that superclass (typically defined from within extensions). Under most circumstances,SQLAlchemy makes this “just work”; when fetching the object from the database, the right subclass will be automat-ically used. However, when creating an object for the first time, HIL must be told which driver to use (is the switcha powerconnect or a nexus?). For these cases, the api calls in question (node_register, switch_register. . . ) will ex-pect a type field to be provided. The module hil.class_resolver provides facilities for finding the appropriatesubclass based on the contents of this type field.

The switch drivers shipped with HIL itself are defined in the modules beneath hil.ext.switches. These includedrivers for the Dell Powerconnect 5500-series (in hil.ext.switches.dell) and the Cisco Nexus 5500 ( hil.ext.switches.nexus), as well as a mock driver useful for testing (hil.ext.switches.mock).

101

https://sqlalchemy.readthedocs.org/en/rel_0_9/orm/inheritance.html


102 Chapter 22. Driver Model

CHAPTER 23

Search

• search

103

Documents

HIL Documentation - Read the DocsHIL Documentation, Release 1.0 1.1Requirements Required software/hardware for running a production HIL include: •Network switches: – At least one