Embed Size (px)
DESCRIPTION
Fermi Gamma-ray Space Telescope. Fermi Large Area Telescope Tutorial on Science Analysis Data Access Tony Johnson Stanford Linear Accelerator Center [email protected]. http://glast-ground.slac.stanford.edu/DataPortal. Data Portal Components. The Data Portal Comprises: Astro Server - PowerPoint PPT Presentation
Citation preview
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 1/18
Fermi Large Area TelescopeFermi Large Area Telescope
Tutorial on Science Analysis
Data Access
Tony JohnsonStanford Linear Accelerator [email protected]
Fermi Gamma-ray Fermi Gamma-ray Space TelescopeSpace Telescope
http://glast-ground.slac.stanford.edu/DataPortal
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 2/18
Data Portal ComponentsData Portal Components
• The Data Portal Comprises:– Astro Server– Common History– Fits Skimmer– Merit Skimmer– Data Catalog
• Web Interface• Linemode Client
– Download Manager• Web Interface• Command Line
– WIRED Event Display• This talk will only cover the items in red.
All components available via the web interface at:http://glast-ground.slac.stanford.edu/DataPortal
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 3/18
““Astro Server”Astro Server”
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 4/18
Astro ServerAstro Server
• Allows data to be selected based on any combination of:– Energy Range– Position in sky (ra,dec,radius)
• Position can be looked up in SIMBAD catalog– Event Class (transient, source, diffuse)– Time Range (MET)
• Can produce FT1 (fits), FT2 (fits) and/or Merit (root)*• Supports multiple “Event Samples”
– Currently “LEOScience” or “Level1”• Can support access to multiple versions of data once
we start reprocessing, or if we want to have any kind of special event selections
• Level1 data currently loaded as soon as it becomes available from pipeline
• Web and command line interfaces available* Coming Soon
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 5/18
Astro Server Web InterfaceAstro Server Web InterfaceYou can give job meaningful name
Leave fields blank for “no-limit”
Expert options allow experimental features to be easily added
Give name of astronomical object, sirius, M31, MCG+02-60-010
Page pre-populated with your most recent selection
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 6/18
Astro Server Web Interface…Astro Server Web Interface…
README contains job parameters
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 7/18
Common HistoryCommon History
• Data Portal History can be used:– Show status of all your skimmer/astro server jobs– To access output directory, pipeline details for all jobs– Recover parameters of any job
Tutorial Demo
Click to go to output directory
Click to go to pipelineClick to recover parameters
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 8/18
Common History…Common History…
• Clicking on job name in history recovers parameters– Allows rerunning same or similar job
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 9/18
““Fits Skimmer”Fits Skimmer”
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 10/18
Fits SkimmerFits Skimmer
• Supports merging any FT1/FT2* files in data catalog• Allows “fcopy rowfilter” style cuts to be applied to FT1 columns
– Example:• ZenithAngle<105 && CTBClassLevel==3
– http://heasarc.nasa.gov/lheasoft/ftools/headas/rowfilter.html• Allows selection of data based on meta-data in data catalog
– Examples• sIntent==“nomSciOps”• LEOScience=="1835Survey“• nMootKey==2103 || nMootKey==2104
• Outputs FT1 and FT2* files
* Coming Soon
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 11/18
Fits Skimmer Web InterfaceFits Skimmer Web InterfaceYou can give job meaningful name
Page pre-populated with your most recent selection
Row Filter
Data Selection (this is the selection Richard has been using for making merged
FT1/FT2 files)
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 12/18
Fits Skimmer Web Interface…Fits Skimmer Web Interface…
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 13/18
Fits Skimmer Fits Skimmer vs.vs. Astro Server Astro Server
• Fits Skimmer– Allows cuts on any FT1
column– Suitable for merging many
files with minimal cuts– Can output FT1, FT2*– Can be used with any
dataset in data catalog– Fully reads all input files
• Astro Server– Allows cuts on energy, time,
position, event class– Suitable for selecting limited
number of events– Can output FT1, FT2, Merit*– Can only be used with data
loaded into database– Uses database for efficient
access to selected events
* Coming Soon
• In future smarter single web interface may automatically choose most appropriate backend tool.
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 14/18
““Data Catalog”Data Catalog”
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 15/18
Data CatalogData Catalog
• Allows access to many datasets including: Level1 data, ASP data, Reprocessed data, MC data sets
Key locations allow attention to be drawn to important datasets
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 16/18
Data Catalog…Data Catalog…
• Data Catalog has been designed to be extensible• Can science groups make better use of data catalog?
– Use it to define standard datasets?• As alternative to sending e-mail with location on user disk• See LEOScience folder as example
– Use meta-data to keep track of common data selections• E.g. goodData==true instead of complex set of sIntent cuts
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 17/18
StatusStatus
• Working on improving documentation– Adding FAQ’s at:
• https://confluence.slac.stanford.edu/display/ds/Data+Access+FAQ– Please contribute your own questions (we will try to provide the answers)
• Data Portal is now mostly complete– Most of the planned functionality is available
• Documentation is improving – We need to work with Science Groups to make sure we are
making optimal use of the available tools– We need input from Science Groups (=you) on what
additional features are needed/desired• E-mail me, or use [email protected]
Fermi LAT Collaboration Meeting, September 2008
T.Johnson 18/18
Discussion ItemsDiscussion Items
• Astro server– What data should be loaded?
• Currently loaded with all FT1 (including LEO) satisfying “Richard’s” sIntent cuts – Should we remove early data?
– Is this is the “right” set of data?
• Currently all “transient, source, diffuse” data classes– Original plan was for less data, may need to beef up server
as we get more data
• Weekly summary DST files?– What data, when/how to make them?
• Can science groups make better use of data catalog?– Use it to define standard datasets?– Use meta-data to keep track of common data selections
