NERC Environmental Big Data Capital Bid Neil Lonie & Andrew Brooks

NERC Environmental Big Data Capital Bid

Neil Lonie & Andrew Brooks

26 September, 2013 2

Dundee Satellite Receiving Station (DSRS) Overview

• Based in Space Tech. Centre, Univ. of Dundee – Director Prof Steve Parkes• Operational since 1976 – Part of NERC Services & Facilities group – Initially

funded via grant then contract since 1978• Worked in partnership with Plymouth Marine Lab. Remote Sensing Group

for many years – Also NERC S&F funded• Formalised this in 2006 – Established NERC Earth Observation Data

Acquisition and Analysis Service (NEODAAS)

• DSRS primary activities – direct satellite data reception, archiving and distribution to support UK environmental research

• Receive from various EO satellites – primarily polar orbiting with Direct Broadcast capability, e.g. NOAA/MetOP-AVHRR and Terra/Aqua-MODIS

• Currently receive around 30 overpasses per day – Archive of over 160,000 passes back to 1978

• Also receive geostationary satellite data giving full global coverage• Operate 5 tracking antennas for polar and 2 fixed antennas for

geostationary data reception


• Raw received data are processed to and distributed in standard low-level formats, e.g. NOAA Level 1b

• Imagery extracted from raw data are also provided in standard image formats and map projections

• Higher-level processing and data analysis generally undertaken by NEODAAS facility partners at PML or users

• NEODAAS-Plymouth provides products derived from Dundee data plus NASA and ESA global data sets

• Products include– Sea-surface temperature, ocean colour, front maps, time composites– Supplied in near-real time or from archive

• NEODAAS user community extends across most NERC science areas – e.g. Marine, Atmospheric, Earth, Terrestrial, Polar

DSRS Overview (contd.)


DSRS Facilities


What is the asset?• Our bid is for a computer hardware asset comprising three main elements:

– 8 Data Processing server computers– 2 Web server computers– 2 Uninterruptible Power Supply (UPS) units

• Data Processing servers– Will significantly enhance our data processing capability– Enable more efficient processing of raw data we receive to user products– Provide resources to increase range of products we can make available to users– For both new data in near-real time and our archive data

• Web servers– Will replace ageing servers used at present– Provide user access to and distribution of products– Improve capability to support wider range of products, increased demand and more users– Provide resources for longer-term redevelopment of website

• UPS units– Ensure stable mains power supply to the servers– Protect against mains power glitches and longer outages


How asset will be used and by whom• Asset to be integrated into our data processing and dissemination

infrastructure• Processing servers will have current and new software installed• They will convert raw data we receive to

– Standard low-level formats– Image products– New higher-level products facilitated by NASA, EUMETSAT etc. processing

software– In near-real time a.s.a.p. after reception

• Also create new products from our archive data• Resulting products to be held on our online storage• Web servers to be used as primary method of dissemination

– Configured to replace old systems for user access to data and products– Website developed to allow new products to be accessed

• Also allow full redevelopment of our website for launch later

10GbE Network Switch10GbE Network SwitchRaw data


How asset will be used and by whom (contd.)

Antenna control

& reception systems

Antenna control

& reception systems

Image Products

Level 1 data

New

Products

Level 1-4 data

Marine Products

Raw received data

Level 0 data

Data processing

system

Data processing

system

140Tb Data Storage Array140Tb Data

Storage Array

LTO Tape ArchivesLTO Tape Archives

DATAUSERSDATAUSERS

NEODAASPlymouthNEODAASPlymouth

Web/FTP servers

Web/FTP servers

L0, L1 & Products

Data & Products


How asset will be used and by whom (contd.)

• DSRS will be primary user – generating data and products and making these accessible

• Our users will be the beneficiaries – improved access and a wider range of products

• Users will include– NEODAAS partners at PML – to derive high-level Marine products in

particular from our data (e.g. SST, ocean colour)– Marine scientists through product access above via PML– Atmospheric scientists through access to products via our website– Terrestrial, Earth and other science areas – new products using

software designed for DB groundstations– International researchers, Education/Training and Public through

direct access to products via our website


Why the resource is needed• DSRS receives data directly from polar orbiting EO satellites• Required to process raw data and provide useable data and images a.s.a.p.

after reception• Rapid access is essential for NRT requirements – Research cruises and

Aircraft campaigns• E.g. cruise ship and FAAM aircraft supported during two campaigns of

NERC ACCACIA programme this year• Started receiving from new MetOp and Suomi-NPP satellites in 2012• These yield higher volumes of data and significantly increase processing

times – 1-2 hours rather than minutes for older satellites• E.g. new NPP data can be received before previous overpass is processed• The resource will enable faster processing of data we receive – Data and

products available to users much faster• Provides service continuity users require


Why the resource is needed (contd.)• DSRS processes data to standard low-level formats and viewable imagery• High-level processing carried out by our NEODAAS partners (primarily

Marine applications) and users in some cases – Requires scientific input/expertise

• Processing software from agencies such as NASA and EUMETSAT is Intended for use by DB groundstations such as Dundee

• Can be used to produce higher-level products without need for scientific input/expertise

• Potential products include – – Atmospheric – cloud top pressure & temperature, total precipitable water– Terrestrial/Earth – reflectance, soil moisture, surface temperature

• The resource provides processing capability to use these software packages for the first time to create a wider range of products

• And allow web server/website upgrade to cope with more products and user demand


Why this addresses Big Data issues

• Again – new satellites have higher data rates and produce more data than DSRS has handled in the past

• E.g. NPP yields data volumes 20x those of Terra & Aqua• Processing times and delay to product availability have

increased significantly as a result• Existing resource limitations prevent higher level processing

• The asset will address these issues by– Enabling more efficient processing of larger volumes of data– Increasing speed from reception to data/image product availability– Provide resources to exploit available processing packages to increase

variety of products


How it fits with Big Data infrastructure

• DSRS data and products could be made available for use within CEMS/JASMIN and the Satellite Applications Catapult

• CEMS/JASMIN facilities and tools might be used for processing and analysis of our data, e.g. entire archive could be processed efficiently for long time-series requirements

• DSRS can become a data provider to the Satellite Applications Catapult, e.g. possible development of applications and services by organisations working with the Catapult

• Initial discussion between DSRS Director and Prof Nick Veck, Satellite Catapult – Interest in DSRS providing data for potential near-real time use


Case for timely procurement

• No open tender delay – cost below OJEU threshold and purchased via existing Uo Dundee supplier contract

• Latest quote indicates under 10 days from order to delivery• Equipment installation and initial configuration – complete

around 6 weeks from funding allocation• Application software installation, migration to new web

servers, initial product creation/testing ~ 6 weeks further• Reach this stage 2-3 months before 31 March 2014 deadline• Begin publicising new products to potential users• More products added – through and beyond March 2014


Response to outline panel feedback

• Generic points – relevance to Big Data as previously indicated– Capability to process larger volumes of data than before– Process these much more quickly – Increase variety of products we can provide

• Specific feedback – none received – panel agreed that proposal – Aims at a large user group– Has strong scientific need

Thank you!

Documents

NERC Environmental Big Data Capital Bid Neil Lonie & Andrew Brooks