Lofar Survey workshop 8/3/2007
Lofar -Surveys and AstroWise
Lofar -Surveys and AstroWise
OmegaCENNOVA – Kapteyn Institute -RuG
8 March 2007Lofar Survey Workshop
OmegaCENNOVA – Kapteyn Institute -RuG
8 March 2007Lofar Survey Workshop
Edwin A. ValentijnEdwin A. Valentijn
Lofar Survey workshop 8/3/2007
“Moore’s law” for N-body simulations
“Moore’s law” for N-body simulations
Courtesy Simon White
Lofar Survey workshop 8/3/2007
Instrument: e-scienceInstrument: e-science
= share resources beyond workstation• Compute power:
– AstroWise grid (RuG, Leiden, MPE, Bonn, INAF)– EGEE-3
• Storage:– AstroWise fileservers (Leiden, RuG, Nijmegen, MPE, Bonn,
INAF)– BiG GRID (RuG- NikHef- SARA)
• Database: all meta + calibration +source data– AstroWise - Oracle
• Application programs + doc’s• Web services• Interoperability: EGEE- EURO-VO
= share resources beyond workstation• Compute power:
– AstroWise grid (RuG, Leiden, MPE, Bonn, INAF)– EGEE-3
• Storage:– AstroWise fileservers (Leiden, RuG, Nijmegen, MPE, Bonn,
INAF)– BiG GRID (RuG- NikHef- SARA)
• Database: all meta + calibration +source data– AstroWise - Oracle
• Application programs + doc’s• Web services• Interoperability: EGEE- EURO-VO
Lofar Survey workshop 8/3/2007
ManipulationManipulation
X-matchX-match
VisualizationVisualization
Direct links to:Direct links to:
CDS Aladin
ALADINALADIN
Lofar Survey workshop 8/3/2007
Multi-tier models for LHC & LOFAR
Multi-tier models for LHC & LOFAR
Large Hadron Collider• Tier-0 (CERN):
– copy of all raw data• Tier-1 (~8-10 centers)
– archive 1/n fraction of raw data & reconstructed data
– regular re-processing of the raw data
– archiving data from Tier-2 centers
– provide central grid services: grid accessible computing and data resources
– support coordination• Tier-2 (~100 centers):
– data analyses– no data archiving
Large Hadron Collider• Tier-0 (CERN):
– copy of all raw data• Tier-1 (~8-10 centers)
– archive 1/n fraction of raw data & reconstructed data
– regular re-processing of the raw data
– archiving data from Tier-2 centers
– provide central grid services: grid accessible computing and data resources
– support coordination• Tier-2 (~100 centers):
– data analyses– no data archiving
LOFAR• Tier-0 (RUG): central processor
– lots of processing, limited archive• Tier-1 (~6-8 centers)
– specialized science analysis– advanced processing of (large)
data sets– dedicated archive, also for Tier-2
centers– provide central grid services:
grid accessible computing and data resources
– support coordination• Tier-2 (~100 centers):
– individual scientists– data analyses– no data archiving
LOFAR• Tier-0 (RUG): central processor
– lots of processing, limited archive• Tier-1 (~6-8 centers)
– specialized science analysis– advanced processing of (large)
data sets– dedicated archive, also for Tier-2
centers– provide central grid services:
grid accessible computing and data resources
– support coordination• Tier-2 (~100 centers):
– individual scientists– data analyses– no data archiving
Lofar Survey workshop 8/3/2007
AstroWise paradigmAstroWise paradigm
“Classical” paradigmForward chaining
Target processing - AweBackward chaining
waterfall model TIER architecture
User hunts upstream
driven by input raw data Driven by query of user
Process in pipelineworkflow
Process in bits and pieces on the flyBackward chaining
Operators push data User pulls data
Results in releases Provide information system
Static archives – publish Dynamic archives –publish Internet
Raw data - obsolete Raw data is sacred
Lofar Survey workshop 8/3/2007
Virtual Survey TelescopeVirtual Survey Telescope
NL:Leiden, Nijmegen, GroningenEU: Napoli, Munchen, Bonn, Heidelberg, Paris Santiago
NL:Leiden, Nijmegen, GroningenEU: Napoli, Munchen, Bonn, Heidelberg, Paris Santiago
raw pixel data pipelines/cal files catalogues
all integrated in one information system100% data lineage
1 dec 2006 delivered !• distributed services
• processing GRID Storage GRID • Methods/services GRID
raw pixel data pipelines/cal files catalogues
all integrated in one information system100% data lineage
1 dec 2006 delivered !• distributed services
• processing GRID Storage GRID • Methods/services GRID
Lofar Survey workshop 8/3/2007
Current Instruments/projectsCurrent Instruments/projects
• VST- OmegaCAM– GT, KIDS, VESUVIO, OmegaWhite, OmegaTrans
• VISTA– UltraVISTA, VIKING via Edinb.
• SuprimeCAM @ Subaru• [email protected], WFC@ING• ACS@HST Coma legacy survey• Panstarrs• Lofar
– Surveys– EoR
• VST- OmegaCAM– GT, KIDS, VESUVIO, OmegaWhite, OmegaTrans
• VISTA– UltraVISTA, VIKING via Edinb.
• SuprimeCAM @ Subaru• [email protected], WFC@ING• ACS@HST Coma legacy survey• Panstarrs• Lofar
– Surveys– EoR
Lofar Survey workshop 8/3/2007
Astro-Wise VO Properties Benefits integrated dynamic db
Astro-Wise VO Properties Benefits integrated dynamic db
• on-the fly re-processing• 5LS: 5 Lines Script• All bits are traced• Administration for parallel processing compute GRID SETI@home• Global solutions –astrometry/photometry• Build–in workflow• Fully user tunable – own provided script• Context: projects/surveys, instruments, mydb• Publish directly in EURO-VO
• on-the fly re-processing• 5LS: 5 Lines Script• All bits are traced• Administration for parallel processing compute GRID SETI@home• Global solutions –astrometry/photometry• Build–in workflow• Fully user tunable – own provided script• Context: projects/surveys, instruments, mydb• Publish directly in EURO-VO
Lofar Survey workshop 8/3/2007
TARGET diagramLofar Survey/ ST ACS
TARGET diagramLofar Survey/ ST ACS
Lofar Survey workshop 8/3/2007
www.astro-wise.orgwww.astro-wise.org
Lofar Survey workshop 8/3/2007
Web services- object makercompute GRID
Web services- object makercompute GRID
Lofar Survey workshop 8/3/2007
Planning 2007Planning 2007
• ADD with Astron - enable communities
• Publish WENSS in VO• connect to EGEE• AstroWise at Astron• Build e-SOC at RC RuG• Source extraction -> source
parameters services • Involve Lofar community
• ADD with Astron - enable communities
• Publish WENSS in VO• connect to EGEE• AstroWise at Astron• Build e-SOC at RC RuG• Source extraction -> source
parameters services • Involve Lofar community