Automating Networks by using API

0

Copyright IDC Frontier Inc. All rights reserved.

Supporting the FutureYour Innovative Partner

0

Automating Networks by using API/Webs-story of a Network Engineer struggling with Programming-

IDC Frontier Inc.UX Development Dept.

Issei Inoue

Feb 24, 2016

11

(C) IDC Frontier Inc. All Rights Reserved.

Agenda

・What Is Network Automation, Network API?

・Case Study (Entering Settings to a Router by Web/APIEntry)

・Software Design for Network API

・Tools Used (Sinatra, MySQL, ActiveRecord, NETCONF, JavaScript, jQuery, etc.)

・Required Skills and Issues

22


What is Automating Networks ?

33


client

HTTPRest API

Sinatra DB

Jobs Setting devices

Workers

Router

Worker

Netconf

Worker

NetconfWorker

Netconf

MySQL

Ruby

CRM

Application

form

【Conventional Network Operation】【Automated Network Operation】

CLI

Router

client

44


What is Network API?

• APIs for network devices

� They can be programatically configured by CLI/GUI, however,

APIs are used to make configuration more easily programmable

• APIs for network services run by operators, etc.

� End users use APIs to set cloud networks such as IaaS

� ex) AWS CreateVpc, AcceptVpcPeeringConnection , etc.

In the first place, what are "network APIs"?

The former is made by manufacturers,

and the latter is made by operators.

55


What is Network Automation

• SDN• Creation of virtual networks on physical networks

• NFV• Virtualisation of network functions on x86 servers

• White box• Switch that separates hardware from software

• Configuration management tools such as Ansible/Chef are used for management

• Automatic configuration by grouping together several

hundred switches (Zero Touch Provisioning)

• Automation of configuration tasks (AutoProvisioning)

In the first place, what is "automation"? The use of software?

This time, this is the

main theme

* Although SDNs or NFVs make AutoProvisioning easierthey are defined as something else here

66


Programming of Tasksclient

HTTPRest API

SinatraDB

Jobs Setting devices

Job registration

Workers

Router

Worker

Netconf

Worker

NetconfWorker

Netconf

Not executed

Job confirmation

Reflection of settings

Tasks done by people are put into

logical patterns and loaded into a

program.

MySQL

Ruby

Process ⇒⇒⇒⇒ API

Information ⇒⇒⇒⇒ DB

+----+-----------+-------------+------------+------+--------+--------+--------------------+----+| id | vpn_num | domain |username | vlan | vrf | zone | segment |job |+----+-----------+-------------+------------+------+--------+--------+--------------------+----+| 72 | NPVN00619 | 70000000619 |issei06192 | 20 | 619 | Henry | 192.168.20.254/24 | 1 || 71 | NPVN00619 | 70000000619 |issei0619 | 19 | 619 | Pascal | 192.168.19.254/24 | 1 || 67 | NPVN00618 | 70000000618 |issei0618 | 618 | 18 | Tesla | 192.168.18.254/24 | 1 || 68 | NPVN00618 | 70000000618 |issei0618 | 617 | 18 | Tesla | 192.168.60.254/24 | 1 || 65 | NPVN00111 | 70000000111 |issei01 | 111 | 110 | Tesla | 192.168.111.254/24 | 1 || 58 | NPVN00615 | 70000000615 |issei0615 | 15 | 6 | Pascal | 192.168.15.254/24 | 1 || 59 | NPVN00615 | 70000000615 |issei0615-2 | 152 | 6 | Tesla | 192.168.152.254/24 | 1 || 64 | NPVN00615 | 70000000615 |issei0615-4 | 154 | 6 | Pascal | 192.168.154.254/24 | 1 || 57 | NPVN00608 | 70000000608 |Issei | 50 | 60 | Pascal | 192.168.1.0/24 | 1 || 63 | NPVN00615 | 70000000615 |issei0615-3 | 615 | 6 | Henry | 192.168.156.254/24 | 1 || 62 | NPVN01615 | 70000001615 |issei1615 | 16 | 15 | Tesla | 192.168.16.254/24 | 1 || 56 | NPVN00612 | 70000000612 |issei0612 | 612 | 612 | Tesla | 192.168.61.254/24 | 0 |+----+-----------+-------------+------------+------+--------+--------+--------------------+----+

there are quite a few

provisioning tools like these

77


Illustration of Linking

APIAPI

PXC(Percona XtraDB Cluster)

netconfnetconf

(Standby)

IP-VPN

(Active)

3 replication

east-regionwest-region

88


99


Tools I used

・Sinatra

・Flask

・Django

Language

・Ruby

・Python

・Go

・PHP

・Bash

DB

・Percona

・ActiveRecord

・MongoDB

・Drizzle

・MariaDB

・Percona

・Redis

Web framework

Configuration

management

provisioning tools

・Ansible

・Puppet

・Chef

・Fabric

Job management

・RabbitMQ

・Celery

・rundec

・cron

・(Bash)

・(MySQL)

Web server

・Apache

・Nginx

・Webrick

・・・・

・・・・

・Web UI

O/R mapping

・Rack

・WSGI

・PSGI

・Netconf

・Bash

・API

・JavaScript

・jQuery

・Ajax

*This is not to say that I recommend these tools.

I might even go so far to say that any tool would be good enough.

1010


API Design

MethodAPI Path

Argument Return Value

Addition of new records Post /privateconnect record Success/Fail

VPN settings Post /privateconnect Job id Success/Fail

Acquisition of information of VRFs to be set Get /routinginstance Job id xml-based VRF information

Acquisition of information of interface to be deleted Get /interface_stats Job id xml-based interface information

Deletion of records Delete /privateconnect_record Job id Success/Fail

Deletion of VPN settings Delete /privateconnect_setting Job id Success/Fail

Acquisition of 10 latest records in JSON format Get /privateconnect.json n/a VPN information in JSON format

APIs and Classes/methods for each action

1111


Tips For Rest APIs

API visibility is also important If possible. I'd like notation methods to be unified, too.

It would be very difficult though.

spinal-case snake-case camel-case dot

How to join - (hyphen) _ (underscore) Uppercase

characters

.(dot)

Examples /v1/idcf-inoue /v1/idcf_inoue /v1/idcfInoue /v1/idcf.inoue

Main Corporation

That Apply TheseGoogle

LinkedIn

Facebook

Twitter

Instagram

slack

YouTube

CloudStack

OpenStack

Facebook

Others Apparently good in

terms of SEO

What's the latest

fashion?

This is often the

case with JavaScript,

jQuery and Ruby.

Not seen a lot

There are several notation methods to connect two or more words.

1212


UI Illustration

1. Extraction of information

2. Addition of settings

3. Deletion of settings

Information targeted in full text searches

can be easily extracted

mysql> select * from private_connects;+----+-----------+-------------+----------------+-------------+------+--------+-----------+--------+--------------------+---------------------+---------------------+----------+| id | vpn_num | domain | account | username | vlan | vrf | bandwidth | zone | segment | created_at | updated_at | job_done |+----+-----------+-------------+----------------+-------------+------+--------+-----------+--------+--------------------+---------------------+---------------------+----------+| 72 | NPVN00619 | 70000000619 | inoue06192 | issei06192 | 20 | 619 | 1000 | Henry | 192.168.20.254/24 | 2015-06-19 10:38:01 | 2015-06-19 10:39:15 | 1 || 71 | NPVN00619 | 70000000619 | inoue0619 | issei0619 | 19 | 619 | 1000 | Pascal | 192.168.19.254/24 | 2015-06-19 10:34:24 | 2015-06-19 10:36:51 | 1 || 67 | NPVN00618 | 70000000618 | inoue0618 | issei0618 | 618 | 18 | 1000 | Tesla | 192.168.18.254/24 | 2015-06-18 04:23:04 | 2015-06-18 06:10:22 | 1 || 68 | NPVN00618 | 70000000618 | inoue0618 | issei0618 | 617 | 18 | 1000 | Tesla | 192.168.60.254/24 | 2015-06-18 06:13:04 | 2015-06-18 06:15:08 | 1 || 65 | NPVN00111 | 70000000111 | inoue01 | issei01 | 111 | 110 | 1000 | Tesla | 192.168.111.254/24 | 2015-06-15 10:42:35 | 2015-06-17 01:58:20 | 1 || 58 | NPVN00615 | 70000000615 | inoue0615 | issei0615 | 15 | 6 | 1000 | Pascal | 192.168.15.254/24 | 2015-06-15 06:51:00 | 2015-06-15 06:51:18 | 1 || 59 | NPVN00615 | 70000000615 | inoue0615-2 | issei0615-2 | 152 | 6 | 1000 | Tesla | 192.168.152.254/24 | 2015-06-15 07:23:34 | 2015-06-15 07:29:34 | 1 || 64 | NPVN00615 | 70000000615 | inoue0615-4 | issei0615-4 | 154 | 6 | 1000 | Pascal | 192.168.154.254/24 | 2015-06-15 10:32:31 | 2015-06-15 10:33:01 | 1 || 57 | NPVN00608 | 70000000608 | Inoue | Issei | 50 | 60 | 1000 | Pascal | 192.168.1.0/24 | 2015-06-12 05:20:00 | 2015-06-15 06:53:06 | 1 || 63 | NPVN00615 | 70000000615 | inoue0615-3 | issei0615-3 | 615 | 6 | 1000 | Henry | 192.168.156.254/24 | 2015-06-15 10:01:53 | 2015-06-15 10:02:54 | 1 || 62 | NPVN01615 | 70000001615 | iinoue1615 | issei1615 | 16 | 15 | 1000 | Tesla | 192.168.16.254/24 | 2015-06-15 09:50:14 | 2015-06-15 09:50:49 | 1 || 56 | NPVN00612 | 70000000612 | inoue0612 | issei0612 | 612 | 612 | 1000 | Tesla | 192.168.61.254/24 | 2015-06-12 05:15:41 | 2015-06-12 05:16:47 | 1 || 73 | NPVN00619 | 70000000619 | %,$,#,&,?,!etc.| issei06193 | 193 | 619 | 1000 | Henry | 192.168.193.254/24 | 2015-06-19 12:32:39 | 2015-06-19 12:36:37 | 1 || 74 | NPVN00619 | 70000000619 | inoue06194 | issei06194 | 0 | 619 | 1000 | Tesla | 192.168.0.254/24 | 2015-06-19 12:39:18 | 2015-06-19 12:39:39 | 1 || 76 | NPVN01619 | 70000000619 | inoue06195 | issei06195 | 195 | 650000 | 1000 | Henry | 172.16.0.254/24 | 2015-06-19 12:45:35 | 2015-06-19 12:45:52 | 1 || 83 | NPVN00624 | 70000000624 | inoue0624 | issei0624 | 625 | 60 | 1000 | Pascal | 192.168.25.254/24 | 2015-06-24 12:23:50 | 2015-06-24 12:24:00 | 1 || 82 | NPVN00624 | 70000000624 | inoue0624 | issei0624 | 624 | 60 | 1000 | Tesla | 192.168.60.254/24 | 2015-06-24 11:09:01 | 2015-06-24 11:12:49 | 1 || 85 | NPVN00626 | 70000000626 | inoue0626 | issei0626 | 626 | 26 | 1000 | Pascal | 192.168.26.254/24 | 2015-06-25 23:55:10 | 2015-06-25 23:55:10 | 0 |+----+-----------+-------------+----------------+-------------+------+--------+-----------+--------+--------------------+---------------------+---------------------+----------+18 rows in set (0.00 sec)

1313





Existing line information is shown

to keep erroneous entry to a minimum

UI Illustration

Static route Next Hop

192.168.10.0/24 192.168.1.100

1414





Erroneous entry is kept to a minimum by displayed

the usage status of the interface targeted for deletion

UI Illustration

1515


Difficult and Important Points to Popularize Automation

• Integration and standardization of input interface� Bash Use of expect, general-purpose in a sense??

� Netconf RPC-based, annotated in XML

� SOAP API Outdated.. Suited to complex input/output?

� Rest API The latest de facto standard URL resource support

• Substantiation of libraries, substantiation of sample code� Netconf is supported on the product side, which is meaningless even if

supported

� Having a mechanism so that Netconf is used is important.

• Software ecosystem comprising network engineers� Even if manufacturers create plugins and libraries, they will not be

popularized if they are not passed onto users as well

1616


Main Libraries Located in Github, Etc.

Juniperhttps://github.com/Juniper/net-netconf

https://github.com/Juniper/netconf-perl

https://github.com/Juniper/netconf-java

https://github.com/Juniper/netconf-php

https://github.com/leopoul/ncclient

Ciscohttps://github.com/jtimberman/ruby-cisco

https://github.com/nickpegg/ciscolib

Brocadehttps://github.com/brocade/ncclient

https://github.com/brocade/brocade （OpenStack Plugin）https://github.com/BRCDcomm/BVC （VyattaController ）https://github.com/zapman449/brocade_switchshow_aliases （Fiber switches）

Alaxalahttps://github.com/sumikawa/netconf

Ideally, there should be standard APIs (APIs that allow all switches share a common configuration.),

however, this would be difficult in a number of ways. I would at least like official libraries from manufacturers

Cumulushttps://github.com/CumulusNetworks/cumulus-linux-ansible-modules

https://github.com/CumulusNetworks/cumulus-linux-chef-modules

https://github.com/CumulusNetworks/net-next

https://github.com/CumulusNetworks/quagga

https://github.com/CumulusNetworks/cumulus-cl-interfaces-puppethttps://github.com/OpenRTMFP/Cumulus （MonaServerusedSW）https://github.com/cotdsa/cumulus

http://cumulusnetworks.com/blog/cumulus-linux-2/

It would be better to register project names(account names) early....φ(・ω・｀ )

1717


Topics and Talk About

Hardships

1818


Setting Netconf on Ruby for JunosNetconf was difficult.

XML format as it is a bit troublesome for human eyes.

However, In Junos, ”show configuration | display xml” is really handy!

When using with Ruby, hyphens are taken as a regular expression,

so it is recommended to use the send method.

RPC is difficult. Sometimes the program cannot get unlocked...

RestAPI is the best . It is easy for beginners.

interfaces {

”interface” {

unit ”vlan” {

description NPVN00615:70000000615:inoue0615;

vlan-id 15;

family inet {

filter {

input 1G;

}

address 192.168.15.253/24 {

vrrp-group 163 {

virtual-address 192.168.15.254;

priority 150;

}

}

}

}

}

}

Junos ConfigXML format

1919


Checking Netconf information on Ruby for JunosShow related information is in XML so the output cannot fit on single screen.

show interfaces xe-2/0/0.111 detail results 3 scrolls

2020


BIG-IP Rest API (Ver11.5 or later)

$ curl -sk -H "Authorization: Basic xxxxxxxxxxxxxx" https://x.x.x.x/mgmt/tm/ltm/node -H 'Content-Type: application/json' -X

POST -d '{"address": "192.168.0.1","description": "testdescription","name": "testname"}'

{"kind":"tm:ltm:node:nodestate","name":"testname","fullPath":"testname","generation":36,"selfLink":"https://localhost/mgmt/t

m/ltm/node/testname?ver=11.5.1","address":"192.168.0.1","connectionLimit":0,"description":"testdescription","dynamicRatio":

1,"logging":"disabled","monitor":"default","rateLimit":"disabled","ratio":1,"session":"monitor-enabled","state":"checking"}[

Node addition

Addition of member to pool

$ curl -sk -H "Authorization: Basic xxxxxxxxxxxxxx" https://x.x.x.x/mgmt/tm/ltm/pool/~Common~pool_test/members -H

'Content-Type: application/json' -X POST -d '{"name": "testname:80"}'

{"kind":"tm:ltm:pool:members:membersstate","name":"testname:80","fullPath":"testname:80","generation":38,"selfLink":"https

://localhost/mgmt/tm/ltm/pool/~Common~pool_test/members/testname:80?ver=11.5.1"}

Setting Sync

$ curl -sk -H "Authorization: Basic xxxxxxxxxxxxxx" https://x.x.x.x/mgmt/tm/ltm/pool/~Common~pool_test/members -H

'Content-Type: application/json' -X POST -d '{"name": "testname:80"}'

{"kind":"tm:ltm:pool:members:membersstate","name":"testname:80","fullPath":"testname:80","generation":38,"selfLink":"https

://localhost/mgmt/tm/ltm/pool/~Common~pool_test/members/testname:80?ver=11.5.1"}

2121


libraries that were useful

Ruby NetAddr PackageProgramming broadcast addresses or gateway addresses (end addresses) from scratch was

more difficult than I expected.

For example, it was surprisingly difficult to programmatically generate the gateway address

192.168.1.254 from the specified value from 192.168.1.0/24 using just the IPAddr Class.

I have solved the problem by using the NetAddr library.

https://rubygems.org/gems/netaddr/versions/1.5.0

http://www.rubydoc.info/gems/netaddr/1.5.0/NetAddr

netaddr1 = NetAddr::CIDR.create('192.168.1.0/24')

# broadcast address calculation

bc_address = netaddr1.last

# Convert to numerical form and subtract 1

gateway_int = NetAddr::CIDR.create(bc_address).to_i - 1

# That numerical value is returned to address notation

gateway = NetAddr.i_to_ip(gateway_int)

# /24 section only is extracted

netmask = netaddr1.netmask

p bc_address

# => "192.168.1.255"

p gateway

# => "192.168.1.254"

p gateway + netmask

# => 192.168.1.254/24

various handy methods are available, such as

output in ARPA format or range designation

and IPv6 compatibility

2222


Many other issues…

• What happens when a user enters in non-IP address format?

• What happens when logically invalid items are entered?

• In what format are errors returned to users?

• How is rollback performed when processing ends midway?

• Where and how are operation logs saved?

• What happens when a target device is in the middle of maintenance or malfunctions?

• How do you go about making a redundant system or DR? What about data

maintenance?

• How do you perform maintenance on the code when devices are replaced?

• Generation management, testing, CI (Continuous Integration)

• How do you transfer knowledge?

There is no end to software development...

How far can we anticipate errors?

2323


Summary

Well, it is not easy… But

How would you like jobs been automated?

Would you like networks to be manageable like

applications?

We can explore new services.

We can control network to make it more stable and secure

With Network Automation,

2424


Summary

New things come into existence by sharing various

people's intelligence together.

That is the Internet.

Networks cannot be created by one person,

nor one company.

What is important is connecting people and things.

A next generation Internet may be born

by the combination of networks and software!

2525


Finally, a little about me...

http://qiita.com/inoueissei

https://github.com/inoueissei

https://github.com/netops-coding

https://www.facebook.com/inoue.issei

https://twitter.com/inoueissei

IDC Frontier Inc.

Technology Development Division

UX Development Dept.

Issei Inoue

I'm not very good at disseminating information.

However, I'm going to do my best from here on.

2626


Thank you for your kind attention.

2727

Supporting the FutureYour Innovative Partner
