Upload
doancong
View
214
Download
0
Embed Size (px)
Citation preview
DYNES: Building a Distributed Virtual Instrument Shawn McKee, University of Michigan
Eric Boyd, Internet2 TIP2013, January 15 2013
Shawn McKee
DYNES Talk Overview
• IntroducIon to DYNES and current status • How we deploy, monitor and maintain DYNES • Issues and new features • Conclusion
1/15/2013 Shawn McKee 2
What is DYNES?
A distributed virtual cyber-‐instrument spanning about 40 US universiIes and 11 Internet2 connectors which interoperates with ESnet, GEANT, APAN, US LHCNet, and many others. SynergeIc projects include OliMPS and ANSE. Dynamic
network circuit provisioning and scheduling
1/15/2013 Shawn McKee 3
Uses For DYNES
• In the LHC • In other leading programs in data intensive science (such as LIGO, Virtual Observatory, and other large scale sky surveys)
• In the broader scienIfic community. • GENI
For regional networks and campuses to
support large, long-‐distance scienIfic data flows and network
research
• The DYNES team is partnering with the LHC and astrophysics communiIes, OSG, and Worldwide LHC CompuIng Grid (WLCG) to deliver these capabiliIes to the LHC experiment as well as others such as LIGO, VO and eVLBI programs
Broadening exisIng Grid compuIng systems by promoIng the network
to a reliable, high performance, acIvely managed component.
1/15/2013 Shawn McKee 4
DYNES Status
• The DYNES project (an NSF MRI 3 year grant) is in the middle of its last year (ends July 31)
• Last set of sites were included in Dec 2012 • We have deployed most of the DYNES planned footprint (9 more sites out of 47)
• We have developed a significant infrastructure to provision, deploy and monitor the DYNES instrument which I will describe here
1/15/2013 Shawn McKee 5
Reservations"
0
20
40
60
80
100
120
February March April May June July August September October November December
ReservaKons
ReservaIons
IniIal DYNES Challenges
DYNES sites are expected to be mostly autonomous afer iniIal deployment. Certainly when DYNES ends sites must take-‐over
DYNES as an MRI has no “formal” funding for centralized services or applicaIon development (but…we sIll have some services).
Nonetheless, we need to have a way to deploy and if necessary modify system configuraIons to get all sites funcIonal and mostly “hands-‐off” in the long run.
We also need to have a way to determine if sites are funcIonal and noIfy them if not, especially in iniIal stages.
1/15/2013 Shawn McKee 10
DYNES Sofware Components
IDC and FDT systems run ScienIfic Linux 5 or 6 (iniIally 5, now deploying on 6)
Circuit provisioning is done with OSCARS (On-‐Demand Secure Circuits and Advance ReservaIon System) originally v0.5 but now using v0.6
Data transfer with well-‐known open-‐source Fast Data Transfer sofware.
Work underway to integrate OpenFlow capable switches -‐ new firmware will support it on S4810 (see later slides)
Monitor component status with Nagios and custom plugins
All configuraIon files, diagrams and site info kept in Subversion repository
Track network switch configuraIon updates with Rancid
1/15/2013 Shawn McKee 11
Approaches
• Too complex and too centralized. • Who maintains it? Does everyone understand how to use it?
A centralized configuraIon manager
(cfengine, ncm, puppet) was rejected
• Anyone can build or install, updates can be deployed from a yum repository
• RPM post/pre scripts allow some scripIng • Can specify other package requirements in RPM spec
Building a base config into RPMS made sense
• Systems run a cron job which regularly fetches ssh public keys from UM webserver
• HTTP/SSL with verified cerIficate used to assure source idenIty
How to access systems for administraIon?
1/15/2013 Shawn McKee 12
Approaches -‐ Kickstart
• IDC/FDT systems reference certain specific repositories or packages in the kickstart so they come up ready to go, appropriate kernel (FDT uses UltraLight kernel), appropriate base packages.
To quickly get FDT/IDC systems built we
generated site and system specific kickstart files that could be referenced by sites via hmp in the event
that they needed to rebuild a system.
• Batch scripts referenced collecIon of site config files
• Just a fun note: used perl Geo::IP module to set Imezones in kickstarts
These files were created in a batch process (shell/
perl) to be downloaded at install Ime over hmp.
1/15/2013 Shawn McKee 13
Approaches -‐ Switches
• Specify switch MAC and iniIal configuraIon file in DHCP server config, then PXE boot switches
• Batch scripts created site specific switch config files from site config files and placed into appropriate locaIon on our ptp host
Dell/Force 10 switches, like many switches, can be pointed to an iniIal
configuraIon file available over TFTP when PXE booted out
of the box
• Batch scripts package switch config files into dynes-‐base-‐idc RPM
• RPM at install sets up simple DHCP and TFTP servers (not enabled by default) which can be used to repeat the iniIal configuraIon process if a switch is ever replaced
ConfiguraIon files are packaged and installed
on IDC hosts
1/15/2013 Shawn McKee 14
DYNES RPMS • configures core services like logging to DYNES loghost, snmp communiIes, ntp, perfSonar services (owamp), ssh. Also includes many configuraIon scripts.
dynes-‐base
• puts in place site specific config files (same file used to build switch and server configs, now used locally for DYNES sofware config)
dynes-‐config-‐sitename
• specific to IDC. Includes switch configuraIon and docs dynes-‐base-‐idc
• specific to FDT. Requires special kernel repo. Packages script to setup storage post-‐install. dynes-‐base-‐fdt
• requires Nagios RPMS (EPEL repo) and installs public key used by nagios server to run checks. dynes-‐nagios
• Ultralight kernels for FDT dynes-‐repo-‐kernel
1/15/2013 Shawn McKee 15
DYNES Yum repository
ConfiguraIon updates will be automaIcally grabbed by yum update, but sites always have the opIon to disable the DYNES repos and update as they wish.
Example: Afer the iniIal installaIon run we wanted to incorporate Nagios. We packaged our Nagios setup into an RPM and made dynes-‐base require that RPM. Next yum update, all systems were accessible by Nagios.
Fairly low maintenance to maintain
Disadvantage that we have to be careful not to break yum updates with bad dependency specificaIons.
1/15/2013 Shawn McKee 16
ConfiguraIon Scripts
• Runs other config scripts • Run manually afer kickstarIng system/installing RPMS install_dynes.sh
• Installs Dell Yum repository (source for firmware updates, OM sofware) • Sets up Dell OpenManage sofware for CLI interface to hardware (BIOS, Storage Controller,etc) • Updates firmware, configures serngs for AC power recovery, CPU VT
install_dell.sh
• Configures OM sofware to email alerts to DYNES admin list dell_alerts.pl
• Configures Dell Remote Access Controller network and user info (references dynes-‐config-‐site file installed in /etc/dynes by RPMS) idrac6_setup.sh
• Configures RAID-‐0 volume for data caching (runs on FDT only) setup_storage.sh
• Configures bridged network interface, needed by KVM. DYNES IDC controller distributed as VM. configure_net.sh
1/15/2013 Shawn McKee 17
Monitoring the Instrument
• Nagios is well known • Can script nagios checks for more detailed funcIonal status
• Integrated with ‘Nagmap’ to map sites
Though the ideal is to have no central point of service it was decided that we need some way to know how
things are going
• Rancid has “saved” us at AGLT2 a couple Imes • Can store configs to any SVN repository – use web interface to Internet2 repo to reference configs easily
We needed a way to track switch configuraIons for sites in case of
breakage or to restore from emergency
• It’s easy to rack a system and never look at it, email alerts assure we can inform sites of problems
• CLI uIls included in OM are useful
Our installaIon includes Dell OpenManage sofware configured to
send email alerts for system problems
1/15/2013 Shawn McKee 19
Nagios Monitor
1/15/2013 Shawn McKee 21
We setup host-‐groups for each site and by equipment type: switch, FDT or IDC host. Can quickly see where we have issues: host not installed or down, network path intermiment. Goal is to improve higher level tests to verify circuit setup and use between DYNES sites
Nagmap of DYNES InstallaIons hmps://dngs.aglt2.org/nagmap/index.php
(Not shown yet: Princeton, MSU, UNH)
1/15/2013 Shawn McKee 23
Current DYNES Challenges • GeMng stable, reliable services in place
– It takes a significant effort to get the iniIal deployments operaIonal – Sites make changes, someImes without realizing the impact on DYNES – Things break and need to be fixed
• Providing end-‐users with easy (transparent) access to DYNES capabiliKes – DYNES as an MRI doesn’t have development effort to integrate with users
applicaIons – We depend upon interested users to drive this: community support is one way
forward. – Other collaboraIons and research projects will leverage DYNES
• Enabling users to fully uKlize bandwidth reservaKons, at least within DYNES – DYNES FDT storage is equipped with 10G interfaces – ReservaIons are typically 1-‐2 Gbps and TCP behaves badly with a 10G NIC
sending into a smaller circuit – DYNES is exploring RoCE and other RDMA techniques to see if they can help
• Finalizing documentaKon to enable user-‐communiKes – We need to provide detailed informaIon on the exisIng APIs, sofware and
examples – Draf version of user Guide is available for comments (see links at end of talk)
1/15/2013 Shawn McKee 24
DYNES and OpenFlow • When DYNES started Sofware Defined Networking was
“alpha” at best – We would have liked to start out with something like OpenFlow for use in DYNES
• We now have an opportunity to integrate OpenFlow within DYNES by retrofirng some sites with Dell/Force10 S4810 switches and “beta” OpenFlow firmware. – TesIng of OpenFlow and the Open Exchange Sofware Suite (OESS) integraIon was successful at Michigan
– Orders for S4810 switches are in progress with plans to retrofit some sites over the next 2 months
• This should provide a simpler and more inter-‐operable DYNES instrument for the future.
1/15/2013 Shawn McKee 25
AddiIonal DYNES Efforts
• We are also doing a number of addiIonal upgrades and extensions to DYNES – Sites are being retrofimed with a perfSONAR node running perfSONAR-‐PS for tracking network performance
– We are explore RDMA over Converged Ethernet (RoCE) as an opIon to improve the effecIveness of data transfers over reserved DYNES circuits
– Monitoring interfaces to track DYNES circuit use are almost ready to go live
1/15/2013 Shawn McKee 26
Conclusion
• DYNES deployments are nearing compleIon • Our DYNES deployment infrastructure has worked very well. Sites are consistent and generally funcIonal out of the box.
• The monitoring and tracking infrastructure is funcIonal and will help focus our last 6 months of effort
• We have some addiIonal efforts on OpenFlow, RoCE, monitoring and documentaIon that should provide an excellent baseline to hand off to end-‐sites in July
1/15/2013 Shawn McKee 27
Useful DYNES Links
• The DYNES web-‐page at Internet2: hmp://www.internet2.edu/ion/dynes.html • We have a preliminary user docs at: • hmps://spaces.internet2.edu/display/dynesdoc/Home
(Feed back welcome but understand this is a work in progress!)
• Subscribe to the DYNES user mailing list at hmps://lists.internet2.edu/sympa/subscribe/dynes-‐users
• Email quesIons to dynes-‐[email protected].
1/15/2013 Shawn McKee 28
Example Kickstart install url -‐-‐url hmp://mirror.anl.gov/pub/centos/6/os/x86_64/ repo -‐-‐name=Updates -‐-‐mirrorlist=hmp://dynes.grid.umich.edu/dynes/ks/centos6-‐mirrorlist-‐updates repo -‐-‐name=Install -‐-‐mirrorlist=hmp://dynes.grid.umich.edu/dynes/ks/centos6-‐mirrorlist # DYNES repos repo -‐-‐name=DYNES -‐-‐baseurl=hmp://dynes.grid.umich.edu/dynes/repo/el6 repo -‐-‐name=Internet2 -‐-‐baseurl=hmp://sofware.internet2.edu/branches/aaron-‐tesIng/rpms/x86_64/main repo -‐-‐name=EPEL -‐-‐mirrorlist=hmp://mirrors.fedoraproject.org/mirrorlist?repo=epel-‐6&arch=x86_64 # Kernel repo here for FDT only repo -‐-‐name=DYNES-‐kernel -‐-‐baseurl=hmp://dynes.grid.umich.edu/dynes/kernel-‐repo/el6 logging -‐-‐host=141.211.43.110 -‐-‐level=debug skipx lang en_US.UTF-‐8 keyboard us network -‐-‐device eth3 -‐-‐hostname fdt-‐umich.dcn.umnet.umich.edu -‐-‐ip 192.12.80.86 -‐-‐netmask 255.255.255.252 -‐-‐gateway 192.12.80.85 -‐-‐nameserver 141.211.125.17 -‐-‐onboot yes -‐-‐bootproto staIc -‐-‐noipv6 network -‐-‐device eth1 -‐-‐onboot yes -‐-‐bootproto staIc -‐-‐ip 10.10.3.240 -‐-‐netmask 255.255.252.0 -‐-‐noipv6 rootpw -‐-‐iscrypted $1$qeLsd;fsdklj~lsdsdfnotourpasswordreally firewall -‐-‐enabled -‐-‐port=22:tcp authconfig -‐-‐enableshadow -‐-‐enablemd5 selinux -‐-‐disabled firstboot -‐-‐disable Imezone America/New_York ignoredisk -‐-‐drives=sda bootloader -‐-‐locaIon=mbr -‐-‐driveorder=sdb -‐-‐append="rhgb quiet selinux=0 panic=60 printk.Ime=1" # parIIons clearpart -‐-‐all -‐-‐drives=sdb part /boot -‐-‐fstype=ext4 -‐-‐size=500 -‐-‐ondisk=sdb part pv.dynes -‐-‐size=1 -‐-‐grow -‐-‐ondisk=sdb volgroup vg_dynes -‐-‐pesize=4096 pv.dynes logvol / -‐-‐fstype=ext4 -‐-‐name=lv_root -‐-‐vgname=vg_dynes -‐-‐size=1024 -‐-‐grow logvol swap -‐-‐fstype=swap -‐-‐name=lv_swap -‐-‐vgname=vg_dynes -‐-‐size=4096 1/15/2013 Shawn McKee 31
Example DHCP Config # For s4810 BMP (bare metal provisioning) # opIon configfile code 209 = text; # opIon ptp-‐server-‐address code 150 = ip-‐address; # opIon ptp-‐server-‐address 10.1.1.10; # opIon boopile-‐name code 67 = text; subnet 10.1.1.0 netmask 255.255.255.0 { range 10.1.1.200 10.1.1.209; opIon subnet-‐mask 255.255.255.0; default-‐lease-‐Ime 1200; max-‐lease-‐Ime 1200; # opIon routers 10.1.1.10; opIon domain-‐name "local"; opIon broadcast-‐address 10.1.1.255; next-‐server 10.1.1.10; group "local" { # rice S4810 #host rice.local { # hardware ethernet 00:01:e8:8b:09:a6; # opIon configfile "/dynes/switch-‐configs/dynes-‐switch-‐config-‐rice.cfg"; # opIon boopile-‐name "/dynes/images/FTOS-‐SE-‐8.3.10.1.bin"; #} host iowa.local { hardware ethernet 5C:26:0A:F4:F7:6F; opIon boopile-‐name "/dynes/switch-‐configs/dynes-‐switch-‐config-‐iowa.cfg"; } host harvard.local { hardware ethernet 5C:26:0A:F4:F7:5F; opIon boopile-‐name "/dynes/switch-‐configs/dynes-‐switch-‐config-‐harvard.cfg"; }
1/15/2013 Shawn McKee 32