View
222
Download
2
Category
Tags:
Preview:
Citation preview
A RESTful Web Service Interface to the ATLAS
COOL DatabaseShaun Roe
1A RESTful Web Service... (Shaun
Roe, Atlas)CHEP'09 Prague
COOL (see talk by A. Valassi, this conference)
2A RESTful Web Service... (Shaun
Roe, Atlas)
htt
p:/
/lcg
app
.cern
.ch/d
oxygen/C
OO
L/C
OO
L_2
_6_0
/doxyg
en
/htm
l/
CHEP'09 Prague
COOL: A database schema and API designed to be technology neutral (Oracle/MySQL/SQLite)
Data structure is hierarchical:Data entry is to a particular folderfolder, channelchannel and with a specific Interval Of Validity (IOVIOV) and tagtag.=>These form a unique set of coordinates for the datum or resourceresource.
Originally in C++, its methods are exposed in Python,in the ‘PyCool’ module. High Voltage values from
the Detector Control System of Atlas SemiConductor Tracker
A resource is defined by a unique URLURLWhat you ‘do’ to that resource is defined by the http methodhttp method:
POST: Create resource GET: Retrieve resource PUT: Update DELETE: Delete resource
The reply is given via http codes.The format of the answer that you want can be further defined by http headers
REST
CHEP'09 PragueA RESTful Web Service... (Shaun
Roe, Atlas)3
REpresentational StateTransfer
REpresentational StateTransfer
noun
verb
e.g. sending a form
e.g. A browser normally GETs a web page
‘404 Not Found…’
‘Accept: text/html’
Roy F
ield
ing’s
dis
sert
ati
on:
htt
p:/
/ww
w.ics
.uci
.edu/~
field
ing/p
ubs/
dis
sert
ati
on/t
op.h
tm
RES
Tfu
l
Motivation
CHEP'09 PragueA RESTful Web Service... (Shaun
Roe, Atlas)4
Rewind to 2006…
People want data from Cool on the web!Solutions:• Use their own SQL from PHP/Python…• Use a shell script to set up environment, then run the Cool api behind some custom code• Use a cron job to generate local html files..and people want to insert data from the web, too.
Aim: Universal Web Service
CHEP'09 PragueA RESTful Web Service... (Shaun
Roe, Atlas)5
Identify resources with url e.g.http://server/database/schema/dbname/some/folder/path/timespan/0-100/channels/23
Use xml as the data transport format.‘GET’ retrieves the values‘POST’ creates the channel in a folder‘PUT’ updates the channel‘DELETE’ deletes the channel from the folder
Multi-channel uploads/downloads should be possible. DB Modification should be protected.
Additional resources to be matched to a url:Database nodelist, folder description, folder payload definition, list of channels in a folder, list of tags; it should be possible to create and delete databases, folders.
‘Accept: text/xml’
Data format: XML
CHEP'09 PragueA RESTful Web Service... (Shaun
Roe, Atlas)6
What aboutJSON ? (see
later)
<channels server="ATLAS_COOLPROD" schema="ATLAS_COOLOFL_DCS" dbname="COMP200" folder="SCT/DCS/CHANSTAT" since="1226271600000000000" until="1226274125000000000" tag="" version="single”><channel id="138950656" since="1226270581000000000" until="1226274125000000000"> <value name="LVCHSTAT_RECV">193</value> <value name="STATE">17</value></channel><channel id="138950656" since="1226274125000000000" until="1226277728000000000"> <value name="LVCHSTAT_RECV">193</value> <value name="STATE">17</value></channel></channels> Definition of an XSD schema allows
validation of data format on upload, or type-ahead and documentation tooltips in an editor:
Namespaces?
Namespaces?
(oXygen editor)
XML Schema Description
JavaScr
ipt
Obje
ct N
ota
tion
Implementation: CherryPy
CHEP'09 PragueA RESTful Web Service... (Shaun
Roe, Atlas)7
www.cherrypy.orgCherryPy is a Python application server. It is very easy to install and use.The ‘Routes’ module (from Ruby) allows automatic or manual mapping of Python classes and methods to URLs:
d.connect('folder_payload','/:server/:schema/:dbname/*folderPath/payl
oad',controller=restController,action='folderPayload’)
def folderPayload(self, server, schema, dbname, folderPath):
http://myserver.cern.ch/cooldb/ATLAS_COOLPROD/ATLAS_COOL_SCT/DCSP200/SCT/DCS/HV/payload
maps this
to this
The http headers and the methods are also available and can be used to route the URL to the appropriate Python method; other modules for security, caching etc are available.
Examples, curl
CHEP'09 PragueA RESTful Web Service... (Shaun
Roe, Atlas)8
Command line examplesCreate a folder:curl 'http://server/DEVDB10/ATLAS_SCT_COMMCOND_DEV/ACALTEST' -u user:pwd -F definition=@folderdef.xml -X POST
Create channels:curl 'http://server/DEVDB10/ATLAS_SCT_COMMCOND_DEV/ACALTEST/test/gain/channels' -u user:pwd -F definition=@calib.xml -X POST
Insert values:curl 'http://server/DEVDB10/ATLAS_SCT_COMMCOND_DEV/ACALTEST/test/gain/timespan/0-9223372036854775807/channels’ -u user:pwd -F payload=@calib.xml -X PUT
cURL (and libcurl, to use from C, C++, Perl) are standard installations. Other languages have similar libraries: PHP – curl, Java – java.net.url, Python – urllib2 … all are standard installations, so accessing Cool via CherryPy accessing Cool via CherryPy typically does not require any extra setup or libraries.typically does not require any extra setup or libraries.
htt
p:/
/curl
.haxx.s
e/
‘In the wild’ examples
CHEP'09 PragueA RESTful Web Service... (Shaun
Roe, Atlas)9
/** * @author formica */public class CherryPyClient {
private String name = "user.name";private String psswd = "mypass";private String readnodeUrl =
"http://server/ATLAS_SCT_COMMCOND_DEV/ROETEST/Branch/Trial1/payload";private String writenodeUrl =
http://server/ATLAS_SCT_COMMCOND_DEV/ROETEST/Branch/Trial1/timespan/0-100;public String coolGetCherryPy() {
try {URL url = new URL(this.readnodeUrl);java.net.URLConnection conn = url.openConnection();InputStream is = (InputStream) conn.getInputStream();BufferedReader in = new BufferedReader(new InputStreamReader(is));String inputLine = in.readLine();is.close();return inputLine;
} catch (Exception e) {e.printStackTrace();
}return "none";
}
$header=array('Accept: text/xml');//$ch = curl_init();curl_setopt($ch, CURLOPT_HTTPHEADER,$header);curl_setopt($ch, CURLOPT_URL,$url);curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);$format=requested_format();$http_result = curl_exec($ch);if ($format != 'text/xml'){ $dom=domxml_open_mem($http_result);}curl_close($ch);
$header=array('Accept: text/xml');//$ch = curl_init();curl_setopt($ch, CURLOPT_HTTPHEADER,$header);curl_setopt($ch, CURLOPT_URL,$url);curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);$format=requested_format();$http_result = curl_exec($ch);if ($format != 'text/xml'){ $dom=domxml_open_mem($http_result);}curl_close($ch);
JavaJava
PHPPHP
httpHeaders={'Accept':'text/xml'}textData=Nonerequest=urllib2.Request(url,textData, httpHeaders)u=urllib2.urlopen(request)xmlDocString = u.read()return xmlDocString
httpHeaders={'Accept':'text/xml'}textData=Nonerequest=urllib2.Request(url,textData, httpHeaders)u=urllib2.urlopen(request)xmlDocString = u.read()return xmlDocString
PythonPython
eXtensible Stylesheet Language for Transformations
Example: Run List
CHEP'09 PragueA RESTful Web Service... (Shaun
Roe, Atlas)10
Search for runs matching criteria, display results;Uses an ‘Ajax’ request to get the CherryPy XML.
Client-side Local server side
XML is inserted directly in the XHTML page, and associated with an XSLT stylesheet to format it as a table
Plans, conclusion
CHEP'09 PragueA RESTful Web Service... (Shaun
Roe, Atlas)11
Use of CherryPy allowed rapid development of a web service interface to Cool which greatly simplified conditions access and adheres to ‘RESTful’ principles.
It was used during 2008 as the engine for the ‘run list’ query page, for various dcs retrieval services, and for insertion of detector status and calibration data; its adoption is growing as a part of web information collations in the detector subsystems.
Future extensions include extra methods for tag viewing, and more flexible output format (e.g. JSON, zipped format…)
Recommended