ElVis web services and systems

Matt Milano

Mentor - Eliot Feibush

Scientific Graphics

3

Data Monitoring Integrated with Job Monitoring

Web Services

ElVis Software Package

• Display program– 45,000 lines of Java code– 300 classes– Summer ’08: Over 200 updates to the code

• Application Programming Interface (API)– Fortran– C/C++– Java

08/08/08

Contribution – Matthew Milano

• NSTX web service– HTML parsing

• Balloon• Transp/rplot• Global Status Panel• Printing• Preferences.java

08/08/08

NSTX web service

• Completely new feature• Integrating NSTX-generated Graphs with ElVis through eml• Many components• many people• many languages (perl, html, java, idl)

08/08/08

NSTX before ElVis

• Web-interface portal for retrieving graphs

• Completely static• png images of

graphs or datasheets

• Original idea was to have NSTX tools launch ElVis, mentor suggested this “easier” solution – many html implementations existed in ElVis.

08/08/08

• Original web interfaces follows flowchart

• Goal – change as little as possible while still getting all new benefits of ElVis

Webtools overview

IDL graph creator

08/08/08

NSTX with ElVis

• Multiple graph windows• Interactive data

exploration• Compare Signals through

copy and paste• Zoom and save graphs• Print the graph at any

resolution/aspect ratio• Interactive log plots

08/08/08

NSTX challenges• Html forms which use javascript in java – not

good• Java has no javascript support• FormView class supplies rudimentary form

support, but creates inaccessible java elements• Original task – no similar quick how-to's• ElVis help, URL loading all use html• Eml is already loaded through http:// protocol• 95% of task accomplished in 5% of the time – remaining 5% proved onerous

08/08/08

NSTX solutions

• Having a standard format for graphs made writing to ElVis an easy task for the NSTX developers (Bill Davis)

• Create a recursive parsing method• Get root htmlDocument element and iterate through

all children elements• Check tag type (using an element toString) and add all

auto-created java elements to Vector<Element> (a list of objects)

• Catch Button during iteration and overload with custom action listener

• Remove original button action listeners

08/08/08

NSTX – input fields• Iterate through all members of

Vector<Element>• Override FormView class to make

createComponent() publicly visible• Apply Overrided class to all elements of

Vector<Element> with “text” identifiers• Attempt to cast into Vector<JTextField>• Remove all failed (null) attempts

08/08/08

NSTX – action handler

• Add ElVis-generated filename to URL string• Open new UrlConnection to perl server with rendered url string (over

1000 characters)

•Iterate through Vector<JTextField> and

Vector<JTogglebutton>, get text, and assemble into

URL string (to create URL as if $_GET[] variable

had been used)

08/08/08

NSTX – perl serverNSTX – perl server

• Developed by Bill Davis• Perl server runs in web-

accessible directory (/p/fgweb) and

receives requests in the form of GET

• Modified perl server to remove automatically generated names and accept ElVis-generated

names• Moved server, changed output style and directory

08/08/08

NSTX – end result

• Nicely parsed, accessible HTML form turned into Java elements dynamically

• Process can be applied to any HTML form

Pretty picture here

08/08/08

Used at several institutes world-wide.

New physics models being added but still

using legacy plotting code.

Access analysis runs stored in database.

Scripted to formulate graphs of fusion data.

ElVis API sends data to visualization client

with minimal changes to Fortran code.

ElVis: A Portal for Scientific Graphics

E. Feibush, D. McCune, L. Randerson

Transport Analysis Legacy Code

Multiple Graph Windows

1. Visualization Client

5. Compute Services

Control Server

------------- Firewall -------------

Explore Data Interactively

Animation & Whiteboard

Security Panel

Data Monitoring

Verify Input - Monitor Results

Subset of total output is stored as URL.

Client polls the URL, appends new data to graphs.

Animate multiple variables Data highlighter tool displays colored traces in data space for each time step in animation. Zoom & scroll with view.

Compare experiments to simulation data.

Java applet. Runs in browser.

Requests data from server running legacy Fortran application.

Multi-Tier Architecture Distributes

Functionality3. Portal Server

Java & Jython programs import ElVis graph classes and gain all encapsulated methods for interaction & display.

Applications

RequestsGraph

Objects

Credentials

Register sessions. Manage credentials.

Create graphs.

Collaboration among clients.

ElVis

XML

6. Experiment Data Server

Access from anywhere on Internet.

Receives requests from visualization clients. Forwards servlet requests through PPL firewall. Returns graph objects.

Command Line I/O Controller

Master

Pseudo-Terminal

Slave

Certificates stored on remote myProxy server.

Convenient for user: enter name and password in security panel instead of managing files.

MDSplus Database

Special interprocess communication channel sends text between master & slave processes. Master forks slave which execs legacy plotting code.

Buffering of standard I/O prevents trivial Linux pipe.

http://w3.pppl.gov/elvis

Glo

bus

job-

subm

it

Acquired Data

Analysis Runs

Encapsulated Methods

User Files

2. HTTP Web Server

Visualization Servlet

ElVis API

4. myProxy Server

X.509 Certificates

08/08/08

rplot & transp server-side processes• Existing, heavily-used pppl analysis programs which are older

than me.• Rplot and transp utilities run on a separate server and are

accessible with a jython interpreter within ElVis• ElVis itself is not tied to the transp or rplot services – the jython

interpreter must be used to send commands to clterm.c which then executes commands on the server.

• If a user runs a command that interrupts the jython process, server does not receive any new data, and thus keeps defunct processes alive.

08/08/08

End server-side programs

• Transp and rplot can be killed by specific commands• Commands must be sent when an ElVis process interrupts/destroys jython process (ie window close event, exiting ElVis)• Add methods which send commands, call them when action is triggered, thereby sending the signals right as ElVis exits.

08/08/08

Problem

• Clterm and 4 others (out of 5) live• Transp has no problems• Solution:

– Created shell script to kill all old pshrxxx processes.– Sort through all processes, regex matching on date,

execution runtime, and executing user

08/08/08

Balloon web serviceusability enhancements

• Fixed input checking – added a toUpperCase method to remove case-sensitive errors

• Made input checking occur earlier in transp retrieval and refined method to check for each field's accuracy instead of check if the first field was accurate

• Added status panel, allowing the user to see server-side console output.

08/08/08

Balloon: plasma state, GEQ

• Plasma state and GEQ files can also be used for/results in a balloon run – field created but no implementation

• Created utility to copy plasma state/GEQ files• Added utility to ElVis servlet, made generic so

other processes may use it.

08/08/08

Global Status Panel

• Challenge: can we put a JinternalPane in ElVis which contains the console output instead of printing it to the standard output window?

• Can we make it so that other classes can write to it?• Can we also log console output using a similar

method?• There are over 300 classes in ElVis – we don’t want to

have to edit 300 files!

08/08/08

• Use Java logger class to create

new ByteStream PrintStream log

• Redirect System.err and

System.out to new PrintStream

• Create new Preferences.java

entry for saving log, and

remembering log numbers

(max 10 logs)• Allows anyone who experiences

bugs to send us their log files

for troubleshooting

Global Status Panel

08/08/08

Print to pdf/ps

• Feature already existed• Tarun Pondicherry implemented aspect ratios

into main ElVis graph windows last summer• Aspect ratios not properly preserved in

printing – code attempted to cast all aspect layouts to 4:3 or 3:4

• Resulted in unrenderable and unreadable graphs.

08/08/08

Print to ps• Built-in java

libraries make job a little easier

• Create publicly visible variables with the ratio of height to width, modified at every window move

•Inject public ratio variables instead of setting a hard-locked 4:3, define custom orientation based on image size.

08/08/08

Print to pdf - complications

• No built-in java libraries• Not enough time to pursue nuances of open-

source libraries during initial implementation• Bad programming practice in open-source

library – entire class is within constructor, pdf printing treated as a static method rather than an object.

• Constant aspect ratio re-definition – occurs at least 6 times within the class

08/08/08

Print to pdf solution

• Create (the only non-constructor) method to recalculate offsets and resize image

• Continually use this method every time aspect ratio was hard-coded in.

08/08/08

Remembering user selection

• First assignment – worked with Michael Shmulevich

• Have ElVis remember last URL, last opened file, last log file, last printed file, and last saved file

• Used static strings to store data during an ElVis run

08/08/08

Keeping preferences across sessions

• Re-wrote existing file Preferences.java• Changed loading method to load entire file into

string array– Easy to loop through– Slightly more memory efficient than an ArrayList or Vector.

• Created individual save methods for each element– Load into memory and loop through existing file in search

of key (ex. load_file=)– Modify existing entry if found, if none is found append key

and entry to the end of existing file.

08/08/08

• Preferences.java – log preferences• Printing methods – print multiple aspect ratios• Balloon Service – use GEQ and Plasma State, see logged

progress• Rplot and transp – exit cleanly• Elvis servlet – generic copy method• NSTX web tools – completely new feature to request NSTX

shot/signal graphs• In-ElVis console: ability to see console output when

running in applet mode or just without cli

Creation Summary

08/08/08

Acknowledgements

• Eliot Feibush – my direct mentor• Bill Davis – author of original NSTX web tools• Andrew Zwicker, Christine Ritter, James

Morgan – PPPL science education program• Eric Zatz – a key member of the ElVis dev team

Questions?

Ben – API, EML

Mike Movie Making User Enhancements Tutorials

Eric Zatz -CPPG summer undergrad ElVis Mark-Up Language

Matt System & Services

Eliot - Mentor

Doug – MentorCPPG Group Head