New perfSonar Dashboard

Andy Lake, Tom Wlodek

What is the dashboard?

• I assume that everybody is familiar with the “old dashboard”:

• http://perfsonar.racf.bnl.gov:8080/exda• https://perfsonar.racf.bnl.gov:8443/exda• The “new dashboard” is supposed (at very

minimum) to reproduce the functionalities of the old one, but with better code organization

• If possible we would like to add new features: interface to central configuration system, gather gridftp information. Other ideas?

What has changed since Madison?

• Matrix services added• It is possible to create, modify, delete

throughput matrices. Latency and traceroute matrices also added but not fully tested.

• Uploading matrix data from collector still needs some debugging: probably we have some data format inconsistency between collector and data store. Should be resolved soon.

Old/New dashboard - overview

Data Store

Collector API

Collector

PS Host

PS Host

database

user

User API

New dashboard

• Minimum objective: New dashboard should keep all the functionalities of the old one, but have code that is better organized, documented, extensible and scalable.

• Beyond minimim objective: Dashboard should interact with the centralized configuration management. Other ideas, suggestions as what should be included in „beyond minimal objective”?

Proposed structure of new dashboard framework

Data Store

Data Access API

Data Persistence Layer

Database

Display GUI Object config GUI Alarms Authentication Collector Other?User mgmt

Status of the components

• We wrote a formal description of the data objects, Andy Lake maintains a design document, available on request.

• I will describe status of each component and what is expected from it

Central data store• The central store maintains list of known

hosts and services,• It also maintains lists of sites, matrices

and clouds.• So far it has no database back end• So far it does not keep history of services

Central data store

• I plan to add back end database as soon as we are done with debugging matrix services. If you are interested in working on this you are welcome to join, especially if you know hibernate.

• Service history – We need to design and code the service history. Andy had some ideas how to define service record, so it should not take much time to put this on paper and code. Preferably we would like to keep as much as possible of history info in memory and not in db, but I am not sure whether this will be feasible.

Data Store Status

• I keep adding new features to data store almost daily.

• To see current status I refer you to my “data store work progress” document.

Interaction between modules

• Modules talk to the data store by exchanging JSON objects using POST and GET methods

• Data access API is a servlet, responding to GET and POST requests from clients returns JSON objects

• Client programs obtain data from JSON objects. Details follow in the second powerpoint presentation.

Collector

• Collector is a program which runs on remote site

• Periodically it connects to data store and asks „do you have jobs to run?”

• Data store returns list of JSON objects representing the jobs.

• Collector executes the jobs and

• Uploads results back to data store.

Collector

• The code has been written by Andy and he maintains it.

• Collector has been tested to work on primitive services. Matrix services need some debugging, but should be ready soon.

What next?GUI Persistence/Db HistoryMatrix

UsersConfig Gui

Alarms and filters

Authentication

Interface to centralized config management?

Gridftp usage data?

GUI• We need a client to display service

information

• It has been suggested that the GUI can be integrated in OSG display. I am fine with that, but we will some sort of display for the debugging while the system is developed anyway.

• We can have several customized GUI’s.

Status GUI• The status of matrix services – maybe we could

adapt Andy’s code for that.

• The status of primitive services – we do not have candidate code, project can be taken by anyone who wants it.

• Cloud display – integrates site and matrix displays. Volunteers?

• History display: We need to finish history data first. Once ready – volunteers are needed.

Config GUI (needs: back end database and persistence)

• Define, modify and delete objects and services.

• STATUS: developer needed.

Display and Config GUI

• The actual technology to be used does not really matter

• We can have many different GUIs

• Unless you have a very good idea on how to do it I think it would be prudent to follow the layout of the old dashboard, as users know it and they seem to like it.

User Management

• Define, modify and delete user information

• Nice, self contained project good for a summer student or undergrad intern. Must know basic java and preferably have internet programming class.

• It has been suggested that we use existing OSG user management system.

Authentication

Operation URL Servlet Filter

Get services {url}services Get services servlet

Update Service {url}/updateService Update service servlet

DN filter

Build latency host {url}/addLatencyServicesToHost?id={id}

Add latency servlet DN filter

Upload probe results {url}/results Upload results servlet

Host filter

... Etc, etc, ...

We need two types of filters: DN and host

• DN filter: javax.security.cert.X509Certificate

• Compare to list of DN’s of known users

• Host filter: Client host IP is obtained from request context.

• Compared to list of known hosts

Alarms (requires: config GUI, Users)

• Define what criteria have to be met to raise an alarm and who should be alerted

• Current dashboard has alarms for primitive services.

• It is hard to define them for matrix services

• Project is moderately hard for primitive services and hard for matrix services.

Status Filters (not to be confused with servlet filters, discussed earlier!)

• Users should be able to define (via GUI) which lower level inputs (like service statuses) should be combined to obtain higher level status (like: site status).

• Filter definition should be configurable via GUI (requires skill in UI design)

• Filters should be then assigned to objects (sites or clouds)• Not easy, but probably rather interesting project.• I was told in Madison that status filters already exist in

OSG. If yes, we should think how to integrate them with dashboard.

Interface to centralized configuration

• Aaron Brown is about to release the centralized config for testing

• We will have to interface the data store to it.

• As of today we know next to nothing about how the interface should look like

• Very likely it will be a rather big and interesting project, with lots of design work.

Gridftp monitoring

• In Madison it has been suggested that we add gridftp monitoring

• You have to find a way to tell gridftp servers to upload usage data to the data store. This is probably rather hard.

• Once this is solved then we have to agree on format of the gridftp usage data and then define service type “gridftp” in the data store.

• However from the data store point of view it is not hard: a modest amount of fairly routine coding.

Documentation• Andy’s design document – ask Andy for access

• DataStore API and work progress document – ask me for access

• DataStore code - will be in BNL public subversion soon, you will need doe cert to access it

• DataStore javadocs will be available

• Mailing list ps-dashboard-devel-l@lists.bnl.gov• http://perfsonar.racf.bnl.gov:8080/dashboard-1.0-

SNAPSHOT/dump

That’s All

Questions, comments, suggestions?