1 ANASAC Meeting – May 20, 2015

Mark Lacy

ALMA archive status and plans

2 ANASAC Meeting – May 20, 2015

StatusArchive size and data rate

• The NAASC archive now contains about 90TB of data.

• The data rate from Cycles 1 and 2 (and future rate from Cycle 3) is only ~50TB/yr, comfortably below the Full Science specification of 200TB/yr.

• Our data link to Chile is 0.1Gb/s (400TB/yr). Working on making this 1Gb/s, (burstable to 10Gb/s), so capable of transferring a 1TB ALMA dataset in 2.2 (0.22) hrs. This should comfortably outpace even the highest peak data collection rates in Full Science.

3 ANASAC Meeting – May 20, 2015

StatusALMA archive query and request handler improvements since last year

• Request handler now shows data hierarchy.• Download options are more clearly shown to the

user.• Angular resolution is now available both as a search

and returned in the results table. • “Programmatic” (command line) interface now

available (though not documented).• More than ~10 lines of results displayed by default.

4 ANASAC Meeting – May 20, 2015

StatusOther archive-related projects

• Pipeline processing interface (PPI) – phase-I prototype now scheduled for end of August (see B. Glendenning talks).

• Use of NAASC cluster by outside, non-NRAO users being rolled out.

• NA-led ALMA software development programs: – CARTA viewer has prototype.– ADMIT enhanced metadata program (line IDs,

previews, moment map generation etc) working on infrastructure, on track so far.

5 ANASAC Meeting – May 20, 2015

IssuesCurrently being worked

• Results table interface is still not very user friendly and needs tidying (e.g. fewer significant figures for angular resolution; angular resolution is called spatial resolution etc).

• 12m vs ACA vs TP observations need to be distinguished (and TP data recognized).

• Numerous “behind the scenes” issues such as incorrect download logging being fixed.

• Adding a search on largest angular scale (and returning in results table).

• Programmatic interface needs documentation.

6 ANASAC Meeting – May 20, 2015

Response to UC2014

• Is there anything that needs addressing here? Otherwise I will hide this slide.

7 ANASAC Meeting – May 20, 2015

Next 180 Days/Future

• “Data tracker” will begin sending out delivery notification emails automatically. Has already begun sending out emails when data are 1 month from the expiration of their proprietary period.

• Planning has begun for rationalizing the row output in the query results table, with the aim of producing one row per source per member OUS. This is a necessary prerequisite for serving pipeline products.

• Planning is continuing for goal of making individual images available to users once the pipeline is producing them, and for making previews that can be downloaded from the archive interface. (Previews have been prototyped for Cycle 0 data.)

• Archive scientist’s F2F meeting in Santiago next month to explore these and other issues.

8 ANASAC Meeting – May 20, 2015

Example preview

9 ANASAC Meeting – May 20, 2015

Summary

• Archive continues to develop, though pace is slow due to lack of developer and (to a lesser extent) science staff resources .

• Also waiting on pipeline to produce images.• Expect only incremental improvements until the

pipeline is producing images for most data (late 2016?), when we will switch to serving individual images as the main products (raw data will still be available of course).

• Stoehr et al. ALMA archive paper published in SPIE – (arXiv:1504.07354) illustrates our long-term thinking.

10 ANASAC Meeting – May 20, 2015

www.nrao.eduscience.nrao.edu

The National Radio Astronomy Observatory is a facility of the National Science Foundation

operated under cooperative agreement by Associated Universities, Inc.