FIONA Update

NRP WorkshopBozeman Montana

August 7, 2017

John GrahamSenior Development Engineer

Calit2/Qualcomm Institute, UCSDjjgraham@eng.ucsd.edu

FIONAs and FIONettes – Flash I/O Network Appliances – Are DTNs:Linux PCs Optimized for DMZs over Distance

FIONAs Are Science DMZ Data Transfer Nodes (DTNs) &

Also Compute/Visualization/ML Nodes

Phil Papadopoulos & Tom DeFantiJoe Keefe & John Graham

FIONAS—40G, $8,000FIONette—1G, $1,000

3U FIONA DTN

● Supermicro 3U Chassis 836BE1C-R1K03B● SUPERMICRO MBD-X10SRL-F Server Motherboard ● 8 Core E5-1680 V4 XEON PROCESSOR 3.40Ghz. ● 128GB RAM● Dual 240GB OS drives● 3.8TB SSD ZFS pool● SAS3 12Gb/s HBA● Dual Mellanox ConnectX-3 Pro EN 40 Gigabit Ethernet Adapter● TPM

https://docs.google.com/spreadsheets/d/1mYYWpwcsAXREER-tiX1KKtpMuFm0hUCY02xggbsQCJM

https://docs.google.com/document/d/1GZqFyAiyZC0SEIeE1TCDv8ypBezLFvsU5ebKuWH0FYk/edit

1U 100G NVMe DTN

● SuperServer 1U 1018R-WC0R with X10SRW-F motherboard● Two Liqid, HHHL, 3.2TB NVMe PCIe Gen3 x8 6.4TB total● 8 Core 3.40G E5-1680 V4 XEON PROCESSOR ● 128GB RAM● TPM● 2X240GB SSD BOOT● 100G NIC

https://docs.google.com/spreadsheets/d/1mYYWpwcsAXREER-tiX1KKtpMuFm0hUCY02xggbsQCJM

https://docs.google.com/document/d/1GZqFyAiyZC0SEIeE1TCDv8ypBezLFvsU5ebKuWH0FYk/edit

2U 100G NVMe DTN

● Supermicro CSE-829HE1C4-R1K02LPB ● X10SRW-F motherboard● Two Liqid, HHHL, 3.2TB NVMe PCIe Gen3 x8 6.4TB total● Dual 8 Core 3.40G E5-1680 V4 XEON PROCESSOR ● 128GB RAM● TPM● 2X240GB SSD BOOT● 100G NIC

● Can add 120TB of SAS3 drives

https://docs.google.com/spreadsheets/d/1mYYWpwcsAXREER-tiX1KKtpMuFm0hUCY02xggbsQCJM

https://docs.google.com/document/d/1GZqFyAiyZC0SEIeE1TCDv8ypBezLFvsU5ebKuWH0FYk/edit

Globus Connect Server

THREDDS access and viewer options

THREDDS access and viewer options

LINK

UC-Jupyter on Comet with CILogon

JupyterHub authenticates a user with CILogon and spawns kernels on CometWe use a Trusted Platform Modules (TPM) on the JupyterHub FIONAs to secure the keys we get from the CILogon member organization.These keys are used to connect jupyter.calit2.optiputer.net to comet.sdsc.edu.

“JupyterHub iPython notebook” (March 12 )Email to Larry describing iPython update to multi user JupyterHub

Andrea Zonca from SDSC “Run Jupyterhub on a Supercomputer” ( April )http://zonca.github.io/2015/04/jupyterhub-hpc.html

Fernando Perez visits UCSD ( May 11-12 )UC-Jupyter Meeting UCSD@BIDS Lab ( June 19 )

“IPython/Jupyter notebook setup on SDSC Comet” ( September )http://zonca.github.io/2015/09/ipython-jupyter-notebook-sdsc-comet.html

Min Ragan-Kelley from UC-BerkeleyAndrea Zonca from SDSC “CILogon module for the Jupyter OAuthenticator” ( October )https://github.com/jupyter/oauthenticator

UC-Berkeley added to the list of CILogon organizations !

https://github.com/zonca/remotespawner/wiki/setup-Jupyterhub-Comet-with-CILogonhttps://jupyter.calit2.optiputer.net:9090/hub/login ( now )

UC-Jupyter on Comet using Trusted Platform Modules (TPM)

UCB-Jupyter FIONATesla K80 GPGPU

UCSD-Jupyter FIONATesla K80 GPGPU

Next Generation GPU JupyterHubMulti-Tenant Containerized GPU JupyterHub

Running Kubernetes / CoreOSEight GTX-1080 Ti GPUs

~$13K Dual 8 core 2.1Ghz CPU, 32GB RAM, 6 480GB SSD, 2x NVMe bays, 40G ConnectX-4, Dual 10G ports, TPM

Ceph

Jupyter

Multi-Institution, Hyper-Converged ScienceDMZ

PATTERN-LAB

100G CephDTN

SDSU

MicroCloudDTN

100G CephDTN

UCSC

100G CephDTN

SDSC

Rook/Ceph - Block/Object/FSSwift API compatible with

SDSC, AWS and Rackspace

KubernetesCentos7

Jupyter

8 GPU Jupyter

THR

EDD

SZFS

THR

EDD

SZFS

March 2017

100G CephDTN

Caltech

CephDTNEDEX

JupyterSTARLIGHT

JupyterCalit2