Embed Size (px)
Citation preview
Edward Colone, Eric L. Boyd, University of Michigan
●○
■
● U-M Network Overview
● perfSONAR Deployment Challenges
● perfSONAR Deployment Automation
● POC Case Study: perfSONAR SC19 deployment
○ SC19 network
○ Timeline
● Next Steps
2
●○
■3
●○
■
● 8 core perfSONAR Nodes
● 2 satellite campus testpoints
● Multiple 1GE and 10GE mobile diagnostic testpoints
● Esmond Data Archive
● MadDash Dashboard
● pSconfig schedule publisher
● perfSONAR Web Admin
4
●○
■
● Major manual upgrade of a 12 node cluster: perfSONAR 3.5 / CentOS 6 → perfSONAR 4.0 / CentOS 7○ Five UNIX sysadmins○ Two days to complete
● Version control done manually - environment skewed● Hard to coordinate system patching and software upgrade
responsibilities between groups● Staff time cost for node addition scaled linearly with each
node added
5
●○
■
● Testpoints and Toolkits must be registered with Data Archivers
● Dashboards read Data Archivers
● Testpoints, Toolkits, and Dashboards read published schedules
● PWA User account provisioning
6
●○
■
Archiver
pS Maddash
Test Points
Sched. Publisher
PWA Admin
Intermediate UNIX admin / Novice perfSONAR administrator / Full Configuration / 2-6 Test Points
7
●○
■
● perfSONAR authors and supports an official Ansible master Playbook and Roles for component provisioning
● End users bootstraps machines with base OSes, user accounts
● Ansible Idempotency allows for continuously scalable infrastructure
● Ansible is agentless, uses SSH - no extra security overhead
● Open source industry standard solution
8
●○
■
● https://github.com/perfsonar/ansible-playbook-perfsonar
● Simple instructions and examples to manage your entire environment
● Quick and easy bootstrapping for all Roles
● Script to create multiple Inventories
● Abstract individual Roles away from the end user
● No previous Ansible experience required
9
●○
■
● Group Variables allow global definitions for fast mass configuration.
● Per-Host variables take precedence for specific host configuration and tuning
● Host file allows for easy inventory management and resource visualization.
● Inventories can be stored in source control for change management and disaster recovery.
● Multiple Inventories can be easily used for multiple Dev and Prod instances.
●●●
10
●○
■
● https://github.com/perfsonar/ansible-role-perfsonar-archive
● https://github.com/perfsonar/ansible-role-perfsonar-installer
● https://github.com/perfsonar/ansible-role-perfsonar-maddash
● https://github.com/perfsonar/ansible-role-perfsonar-testpoint
● https://github.com/perfsonar/ansible-role-perfsonar-toolkit
● https://github.com/perfsonar/ansible-role-perfsonar-psconfig-publisher
● https://github.com/perfsonar/ansible-role-perfsonar-psconfig-web-admin
psconfig - Web Admin
Installer
Testpoint
Toolkit
Archive MadDash psconfig - publisher
11
●○
■12
Intermediate UNIX admin / Novice perfSONAR administrator / Full Configuration / 2-6 Test Points
Archiver
pS Maddash
Test Points
Sched. Publisher
PWA Admin
Download Playbook / Bootstrap Roles / Configure Inventory
●○
■
Archiver
pS Maddash
Test Points
Publisher
PWA Admin
Download Playbook / Bootstrap Roles / Configure Inventory
pS Maddash
Test Points
Sched. Publisher
PWA Admin
Manual Deployment
Intermediate UNIX admin / Novice perfSONAR administrator / Full Configuration / 2-6 Test Points
13
Archiver
Automated Deployment
●○
■14
●○
■
perfSONAR TestPoints
● Virtual infrastructure included:○ Ansible Controller○ Esmond Data Archiver○ MadDash Dashboard○ JSON Schedule Publisher
Administrative VMs
● Physical Testpoints managed entirely by Ansible● Vendor managed testpoints deployed in booths
seamlessly integrated in to overall system.● Automated Mesh deployment reduced sources of
error and introduced time savings.
15
●○
■
● Standard, project-supported playbooks
● Additional specialized playbooks developed for site-specific enhancements
● Scripts to automate deployment on Ansible controller
● Pre-configured virtual infrastructure before the conference and physical infrastructure on-demand.
● Provisioned all production systems at SC
● Near one-touch deployment
● Used Vagrant as proof-of-concept to test configuration in a virtual environment before the conference
16
●○
■
● Systems required little manual setup
● Minimal OS
● Ansible user account (scripted to minimize errors)
● New systems added to inventory and provisioned as they were brought online
● Allowed testing of partial deployment instead of having to wait for everything
● Adjustments to systems applied consistently
● Significant labor savings left time to work on things needing attention
●
17
Best SCinet perfSONAR deployment in recent years
●○
■
● Ansible support for more optional tools and tests
(GridFTP, SNMP, nmap, etc.)
● Enhanced Vagrant support for virtual development
and proof of concept environments
● More customizations for MadDash, PWA, etc.
● Enhanced automated environment troubleshooting
18
●○
■
● Antoine Delvaux
● Mark Feit
● Ivan Garnizov
● perfSONAR Development Team
● U-M Networking Staff
● Katarina Thomas
19
