03/17/2014

Data Management System -Data Services-

•Temporary Experiment Data Brief•Data Services News•MUSTANG update- QC on PASSCAL Data

PASSCAL SC Spring 2014 1

Temporary DataIs 26% of DS Holdings

03/17/2014 PASSCAL SC Spring 2014 2

Temporary Experiment DataIs Large, Important

03/17/2014 PASSCAL SC Spring 2014 3

_PASSCAL Virtual Network 9577 Stations to date

03/17/2014 PASSCAL SC Spring 2014 4

http://www.iris.edu/gmap/_PASSCAL

Notable DS Activities:

03/17/2014 PASSCAL SC Spring 2014 5

• The Livermore Auxiliary Data Center is operational• Currently being used by LLNL staff to access data using existing web services, was delivered to them on time• This ADC is used like its locally located on the same LAN, since it has 10Gb connectivity:

• Archiving of data automatically multicasts to both LLNL and local storage in Seattle; no need to cache

• Can service breq_fast and web service requests, not currently done routinely. (just weekly as function test)

• We anticipate installing a global load balancer to accommodate traffic to be sent to ADC when load is high in Seattle

Notable DS Activities:(Cont’d)

03/17/2014 PASSCAL SC Spring 2014 6

• Data Services has begun integrating a 10% budget decrease

• We will migrate large, write-once read never data like PASSCAL Flex Array “RAW” data to lower-cost tape for example.

• I am currently undergoing an extensive audit to prioritize storage strategies, perform de-duplication, etc

MUSTANG- QC Across All Data

03/17/2014 7PASSCAL SC Spring 2014

What does MUSTANG Stand For?

I’ll only say this once: MUSTANG is an acronym violation that stands for:

Modular Utility for Statistical kNowl- edge Gathering

03/17/2014 8PASSCAL SC Spring 2014

Data CoverageNetwork Start End # Records

Per Metric

_GSN 2010-02-26 2013-03-12 3.2 M

_PASSCAL 2007-01-01 2013-11-02 225 K

_OBSIP 2011-11-23 2012-05-20 103 K

_CASCADIA 2011-01-04 2013-03-12 125 K

TA 2013-01-07 2013-01-16 278 K

II 2011-01-04 2013-03-10 580 K

IU 2010-02-27 2013-03-10 789 K

(representative)

Latency measurements (all networks) are currently working on >36,000 channels!

http://www.iris.edu/files/MUSTANG/reports/_PASSCAL.metrics.txt

03/17/2014 9PASSCAL SC Spring 2014

March 2014: Decision to use Livermore (ADC) for MUSTANG

Since FDSN web services are currently installed and running at the offsite Auxiliary Data Center at LLNL, we’ll utilize the server and storage VM environment and update the RDBMS in Seattle simultaneously.

This will act as a test bed, but in addition will offload Seattle resources which are currently at maximum I/O

03/17/2014 10PASSCAL SC Spring 2014

Accessing MUSTANGMetrics

There will be a “live” web service front end soon: http://service.iris.edu/mustangbeta/measurement

s/1 is main landing page, with help

A URL builder to help construct the query and get correct syntax is directly reached using:

http://service.iris.edu/mustangbeta/measurements/docs/1/builder/

03/17/2014 11PASSCAL SC Spring 2014

Quick Look at the Interface; Similar to all current web services:

03/17/2014 12PASSCAL SC Spring 2014

Available Soon: “Visualizing” Metrics There is a beta version of a web service front end

that can access the stored metrics that will be similar to this unreleased version:

http://mazamascience.com/MUSTANGDatabrowser/

03/17/2014 13PASSCAL SC Spring 2014

We are committed to PH5 Format Metadata Exposure

We will enable parsing of station/site metadata that is currently stored in PH5 so that utilities like http://www.iris.edu/gmap can display locations and enable increased awareness

NOTE: This is for PH5 data sets only and will not include “assembled” data sets, as these are in no standardized or parse-able format

03/17/2014 14PASSCAL SC Spring 2014

We have begun the migration out of Oracle RDBMS We have just entered into an agreement to

migrate our 2.1Tb Oracle RDBMS to Postgres. (EnterpriseDB)

We will not renew FY15 Oracle in October but can still use it, without support, so we are in a hurry

We currently have a solid list of known unknowns and workarounds, and Oracle-specific procedures that we’ll have to work on.

Consider this a traveler’s advisory, but we intend it to be transparent externally

03/17/2014 15PASSCAL SC Spring 2014

We have begun the migration out of Oracle RDBMS (cont’d)

We will have support and professional help We currently have 9 Postgres databases in

operation so we have PL/PGSQL tribal knowledge that we can leverage.

Not totally new, but PostgreSQL dialects are different.

We hope to have it majorly done in 3 months

03/17/2014 16PASSCAL SC Spring 2014

That’s all for now

Questions? Comments? Requests?

03/17/2014 PASSCAL SC Spring 2014 17