From Photons to Petabytes: Astronomy in the Era of Large Scale Surveys and Virtual Observatories

From Photons to Petabytes:Astronomy in the Era of Large Scale Surveys and Virtual Observatories

From Photons to Petabytes:Astronomy in the Era of Large Scale Surveys and Virtual Observatories

R. Chris Smith

AURA/NOAO/CTIO

“Classical” (Optical) Astronomy“Classical” (Optical) Astronomy• 1-4 investigators propose for telescope time• Obtain 1 to 5 nights, or 1 to 5 hours!

Oversubscription on largest telescopes (e.g. Gemini!) severely limits time per investigator

• Travel to distant telescope siteor not: Remote Observing, Service Observing, Queue

• Observeor not: clouds (OUCH!)

• Take 5 to 50 GB of data home (on tapes)• Reduce & Analyze “by hand”

Extract every detail from those bits Often takes months per night of data

Optical WindowsOptical Windows

QuickTime™ and aMotion JPEG A decompressor

are needed to see this picture.

Can “classical” techniques answer the BIG questions?Can “classical” techniques answer the BIG questions?

• Where do we come from?Star Formation, Nucleosynthesis

• Are we alone?Proto-planetary disks, search for planets

• Where are we going?Big Bang & the Expansion of the Universe

• What is the Universe made of?What types of matter? What types of energy?

Today’s BIG Questions:Dark Energy & Dark MatterToday’s BIG Questions:Dark Energy & Dark Matter

Dark Energy is the dominant constituent of the Universe.Dark Matter is next.

95% of the Universe is in Dark Energy and Dark Matter, for which we have little or no detailed understanding.

1998 and 2003 Science breakthroughs of the year

Brief History of Dark EnergyBrief History of Dark Energy

• 1990s

Wanted to measure the DECELERATION of the Universe

Use SUPERNOVAE as cosmic yardsticks

(Pete Challis, Harvard-Smithsonian Center for Astrophyics

Hubble Space Telescope, NASA)

Type Ia Supernovae are powerful cosmological probes

Distances to ~6% from brightness

Redshifts from features in spectra

Type Ia SNe: Type Ia SNe: One Parameter One Parameter

FamilyFamily

StandardizableStandardizableCandlesCandles

(Supernova Cosmology Project, Kim et al)

~ 0.13 mag

Color

Rate of decline

Peak brightness

-- gives --

δd/d ~ 0.06

An Accelerating Universe?!An Accelerating Universe?!

Riess et. al. 1998

A Repulsive ResultA Repulsive Result

• Expansion of Universe is accelerating?

• Implies something WRONG?Two independent competing teams: SAME result

Some sort of dust? Evolution?

“Higher-z” SN team:8 new SN @ z>1.0“Higher-z” SN team:8 new SN @ z>1.0

Riess et al, ApJ, 2004Riess et al, ApJ, 2004

A Repulsive ResultA Repulsive Result

• Expansion of Universe is accelerating!!!

• Implies something NEW!

• Regions of empty space REPEL each other!

“Cosmological constant”?

• Einstein’s greatest blunder… OR NOT?!!

Something going on in the vacuum?

What is Dark Energy?What is Dark Energy?

New Fundamental Physics!

G + f(g) = 8G [ T(matter) + T(new) ]

????

Two philosophically distinct possibilities:● Gravitational effect, e.g. Cosmological Constant, or

gravity “leaking” into extra dimensions● A “Vacuum energy” effect, decaying scalar field

Sociology of Dark EnergySociology of Dark Energy

• Dark Energy may be pushing the universe APART

• But it is pulling the Astronomy & Physics communities TOGETHER HEP interests in fundamental physics HEP experience in large datasets

Attacking the Question of Dark EnergyAttacking the Question of Dark Energy• “Classical” approach won’t work

Not enough telescope time Difficult to control calibrations & systematics

• LARGE SURVEYS Goal: Provide large, uniform, well calibrated,

controlled, and documented datasets to allow for advanced statistical analyses

Control calibrations & systematics to <1% Larger collaborations provide both manpower and

diverse expertise• Including both traditional astronomers and high-energy

physicists

Dark EnergyROADMAP to understandingDark EnergyROADMAP to understanding• Today

ESSENCE, large international group of astronomers

• Coming Soon to a telescope nearby Dark Energy Survey

• Camera built by Fermilab, majority DOE funding• Data Management System led by NCSA• Groups from Spain and United Kingdom recently joined

• The next BIG step LSST

• Camera built by SLAC, Data Mgmt with NCSA, • NSF + DOE funding, also inc. LLNL, Brookhaven, others

• Stepping UP Space-based work: JDEM (SNAP and/or others)

• NASA + DOE funding

Today:ESSENCE (+SuperMACHO)Today:ESSENCE (+SuperMACHO)• Use a LARGE (~200 SNe), UNIFORM set of

supernova light curves to allow us to study the evolution of the expansion of the universe Constrain “w”, the equation of state parameter of Dark

Energy, to ~10%

• 30 half-nights per year for 5 years (2002-2006)• Use other half of nights to constrain possible

DARK MATTER candidates The ‘SuperMACHO’ project Search the Large Magellanic Cloud for microlensing

Searching for Supernovae (and other transients)Searching for Supernovae (and other transients)

High-z SN Team

The Strategy The Strategy

• Repeatable

• Reliable

• Wide-field

• Multi-color

• Imaging

• CTIO Blanco 4m + MOSAIC II

• Every other night, Oct - Dec, 2002-2006

ESSENCE+SuperMACHOThe data flows…ESSENCE+SuperMACHOThe data flows…• The telescope

CTIO’s Blanco 4m

• The camera MOSAIC 8Kx8K imager (67 megapixels)

• Exposures of 60s to 400s• Collect 20GB of RAW data per night• Data must be reduced and analyzed in near REAL

TIME (within ~10min)• Data ‘Reduction’ = 5x EXPANSION!

Roughly 3TB per year

… and flows… and flows

• MUCH larger data flow than most other astronomical projects

• With ADDITIONAL complication of real-time reduction & alert requirement Must plan spectroscopic follow up on LARGEST

telescopes (Gemini, Keck, VLT, Magellan, …)

• We THOUGHT we were ready A few CPUs (cluster of 20 x 1GHz)A few disks (4 x 4TB “data bricks”)

• But…

ChallengesChallenges

• Moving the data From Chile to the U.S.

• Storing the data Filling up racks with “data bricks” Keeping track of the data Initial database didn’t cut it

• Reprocessing the data Pipeline can keep up with real time flow But need to reprocess past years of data when

improvements are made to software

Coming Soon (2009?):Dark Energy SurveyComing Soon (2009?):Dark Energy Survey• Investigate Dark Energy using 4 complementary

and independent methods Various types of distance measurements, based on

standard luminosities, standard yardsticks, and standard volumes

• Combine the results to provide the best (to date) constraints on the equation of state of Dark Energy

The Instrument:Dark Energy CameraThe Instrument:Dark Energy Camera

Focal Plane:• 64 2k x 4k CCDs

• Plus guiding and WFS• 0.5 GIGApixel camera

• Will be the largest focal plane built to date

The Data:Dark Energy SurveyThe Data:Dark Energy Survey• Each image = 1GB• 350 GB of raw data / night• Data must be moved to NCSA before next night

begins (<24 hours) >36Mbps internationally

• Data must be processed within ~24 hours Need to inform next night’s observing

• Total raw data ~0.2 PB• TOTAL Dataset 1 to 5 PB

Reprocessing planned using Grid resources

Astrophysical ExplorationAstrophysical Exploration

• Wavelength

• Angular resolution

• Area surveyed

• Depth

• Time resolution

Image from DLS & Tony Tyson

LSST: The InstrumentLSST: The Instrument• 8.2m telescope

Optimized for WIDE field of view (FOV)

• 3.5 degree FOV• 3.5 GIGApixel camera

• Deep images in 15-30s• Able to scan whole

sky every 4-5 nights• Site: either Baja, CA

or Northern Chile

LSST: The Data FlowLSST: The Data Flow• Each image roughly 6.5GB• Cadence: ~1 image every 15s• 15 to 18 TB per night

ALL must be transferred to U.S. “data center”• within <24 hours, 4Gbps internationally• Possibly within image timescale (15s), ~10Gbps

• REAL TIME reduction, analysis, & alerts Send out alerts of transient sources within minutes Provide automatic data quality evaluation, alert to

problems Change survey observing strategy on the fly based on

conditions, last field visited, etc.

LSST: The Data FlowLSST: The Data Flow

LSST: The Data FlowLSST: The Data Flow

DES, LSST, … and now for the REST of the ScienceDES, LSST, … and now for the REST of the Science• Ongoing (ESSENCE, SuperMACHO, etc.)

and future (DES, LSST, etc.) projects will provide PETABYTES of archived data

• Only a small fraction of the science potential will be realized by the planned investigations

• How do we maximize the investment in these datasets and provide for their future scientific use?

The Virtual ObservatoryThe Virtual Observatory

• What is VO?Provides the framework for global access to the various

data archives by facilitating the standardization of archiving and data-mining protocols.

Enables data analysis by providing common standards and state-of-the-art analysis tools which work over high-speed wide area networks

• What is VO not?An organization funded to provide a single universal

archive of all astronomical dataA provider of resources (storage, computation,

bandwidth)

A Global EffortA Global Effort

VO ChallengesVO Challenges

• Provide Access to the Content Multiple distributed archives, some on the

scale of many petabytesArchives provide content, the VO knits those

resources together

• Provide the Standards Allow variety of archives talk to each other Develop generalized data model(s) for

different instruments/different wavelengths

VO ChallengesVO Challenges

• Provide the User Interfaces Streamline data discovery, data understanding,

data movement, and data analysis

• Support the Analysis Support large queries across distributed DBs Support statistical analysis across results

(Grid)

• All the “boring” bits (infrastructure) Security, handshaking, resource management

Chris Miller/NOAO

One Archive’s View of VOOne Archive’s View of VO

NOAO’s Data Products ProgramNOAO’s Data Products Program• Management of data from all NOAO and some

affiliated facilitiesKPNO, including Mayall 4m (MOSAIC, NEWFIRM)CTIO, including Blanco 4m (MOSAIC, ISPI)SOAR & WIYN systems

• Provide access to large volume (TBs to PBs) of archived ground-based optical & infrared data and data productsVirtual Observatory “back end”; CONTENT

• Enable science based upon distributed data and data products, developing tools and servicesVirtual Observatory “front end”; UI and TOOLS

Strategic PartnershipsStrategic Partnerships

• In Local Systems Vendors: Local Storage, Processing, Servers

• In Remote Systems Supercomputer center(s) to provide bulk storage, large scale

processing (e.g. NCSA, SDSC’s SRB) Grid processing, storage

• Connectivity High-speed national and international bandwidth

• Scientific VO Partners to develop standards, provide tools Providing services to, and collecting feedback from, physics and

astronomy user communities Providing strong VO node in South America

• Hi ho, hi ho…

• Back to looking for dark energy and other diamonds in the data mines…