Upload
nakia
View
30
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Skyalert and Event Processing. Roy Williams Andrew Drake, Matthew Graham, Ashish Mahabal California Institute of Technology. This talk defines these. VOEvent : Int’l standard representation of data for astronomical transient as XML VOEventNet : A Protocol for pushing VOEvents, and its nodes - PowerPoint PPT Presentation
Citation preview
Roy WilliamsAndrew Drake, Matthew Graham, Ashish Mahabal
California Institute of Technology
Skyalert and Event Processing
This talk defines these
VOEvent: Int’l standard representation of data for astronomical transient as XML
VOEventNet: A Protocol for pushing VOEvents, and its nodes
Skyalert: Software for managing VOEvents: web, subscriptions, repository
VOEvent Key Concepts
IVORN – unique identifier for event Instance of a Stream that also has IVORN How is it resolved?
Who author’s provenance
What data model designed by author May have URLs … permanence, calibration?
WhereWhen targeting in spacetime
Citation related events “followup", "supersedes" or "retraction”
3
VOEventLib
http://lib.skyalert.org/VOEventLib/
Python interface to VOEvent Code built automatically from schema
With ‘utility’ package Vutil
v = VOEvent.VOEvent(role='test', ivorn='ivo://testing#111') author = Author(contactName=['Donald Duck'])v.set_Who(Who(Author=author))
what = What()what.add_Param(Param(name='apple', value='123'))what.add_Param(Param(name='orange', value='124'))v.set_What(what)
s = stringVOEvent(v)print s
VOEventNet Key Concepts
Broker Broadcaster of received events May know which subscriber wants what
Subscriber Connects to Broker Long-term connection
Publisher Scientifically responsible Works with a broker May sign events with PGP
5
VOEventNet
a network of independent nodes exchanging VOEvents
with the VOEvent Transport Protocol (VTP)
LOFAR and 4piSkyrunning CometNASA events
running GCN/TAN
DC3.comrunning Dakota
Skyalertrunning Comet
CBATAAVSOCRTS1,2,3CSS NEO
VOEventNet defined
http://arxiv.org/abs/1206.0236
query
subscribe
relay
annotate
translate
translate
registry
publish
repository
VOEventNet Patterns
Query
VOEventNet
External annotator
Translate
Translate
Publish
repository
SkyalertOpen sourced VOEventNet node
Push messagesPush actionsWeb interfaceWeb services
Many types of events can be homogenized and interoperatedSupported by US National Science Foundation OCI-0915473
SkyalertAnnotation
Annotation can be:
• Newer observations or calibration
• Archive search for matches
• Machine-learning ‘decision’
• Follow-up observation
• Comment / Circular
VOEventNet
PublishAnnotation
Annotation
How can annotations be queued and clouded in case of high event rate?
Two lists of events
betelgeuse.ligo.caltech.edu
All ‘events’ from five high-energy missionsincluding ‘utility’ events
Received by VTP
skyalert.org
‘role=observation’ (astrophysics) events onlyXMPP and email translations
Skyalert Stream Registry... will be VO registry
http://www.skyalert.org/streams/
12
Skyalert Key Concepts
Stream Event semantics defined by author
What does Burst_SOD mean in the Fermi stream? Like a data dictionary, a schema, common metadata
Portfolio Event + associated data
As collection of VOEvents with the observation first
Query Define what is ‘interesting’ For Past or Future events Language, as XML, SQL tables (flex and fixed), KW dict,
triple store??
13
Concept: Stream as Event Template
Event author defines parameters and meaning in advance.stream event
made byperson in daytime
made byrobot system at night14
SWIFT portfolio
Notices from the satellite Circulars from the humans
15
Event DatabasesTechnology and Language Evaluations
Stardog SELECT ?s WHERE {?s voe:hasParam ?p . ?p voe:hasName "…"^^xsd:string . ?p voe:hasValue ?v .
FILTER (?v < …)}
MySQL fixed: with a separate table for each stream select id from … where … < …
MongoDB (NoSQL): ivorn, param name, value db.test.find({"param":…, "value":{"$lt":…}})
MySQL flex: with a params table select id from params where name = "…" and value < …
eXist native XML for $event in /voe:VOEvent where $event/What//Param[@name = '...']/@value < "..." return
<id>{$event/@ivorn}</id>
eXist XML
MySQL flex
Mongo noSQL
MySQL fixed
Stardog triples
0 10 20 30 40 50 60 70
Speed
Skyalert web servicesfrom django.utils import simplejsonimport urllib
sk = http://skyalert.org
# CRTS and CRTSCircular["First"]["eventClass"] == "Supernova"query = "285"
jtableurl = "%s/events/jtable/%s/" % (sk, query)
while 1: jsontable = urllib.urlopen(jtableurl).read() pytable = simplejson.loads(jsontable) list = pytable["list"] print "Found %d portfolios" % len(list) for pf in list: ivorn = pf[1]
dict = {} dict['ivorn'] = ivorn dict['JSON'] = 'on' params = urllib.urlencode(dict) resolveurl = "%s/resolve/" % sk jsondetail = urllib.urlopen(resolveurl, params).read() pydetail = simplejson.loads(jsondetail) # ivorn of the primary event of the portfolio should be here crtsevent = pydetail["portfolio"][ivorn] mag = crtsevent["FirstDetectionparams"]["magnitude"]
tok = ivorn.split('#') localivorn = tok[1] date = crtsevent[""]["ISOtime"] print localivorn, date, mag
jtableurl = jsonresult["next"] if jtableurl == "finished": break
1107030071174105491 2011-07-03T08:36:01 18.9354001107030091124153498 2011-07-03T08:01:30 18.7563991107031180794128821 2011-07-03T04:39:47 20.1698001107031070764142675 2011-07-03T04:07:39 18.1661001107021180094118167 2011-07-02T10:46:06 18.6014001107021041264109182 2011-07-02T09:31:06 15.8796001107011150104107901 2011-07-01T10:47:36 18.2796991107011120104119909 2011-07-01T10:46:15 17.8748001107010040834139996 2011-07-01T06:17:32 18.4836011107011040674116917 2011-07-01T04:13:08 19.630899
Magnitudes of CRTS supernovae
17
Who are the subscribers? What size telescope / Data mining / Citizens Is it the same as the “GCN followup” gang? How can we set up a clearinghouse of information?
How to build “simulated event stream” for LSST? So prototypes can be tested Remember average rate and burst rate
What are annotation services for LSST? Same as “common queries”
What is the security requirement? In testing, commissioning, observing
What language for alerts and history queries Triples, SQL, noSQL, XML, ...