SSEC Satellite Data Services RabbitMQ Messaging System · CADU Satellite Data Services GOES-16 Data Flow SDI sdigrbbeta SATBUF1 rsync GRBGAMMA arcwrite2 csppbeta GEOCAT/AIT Level

SSEC Satellite Data ServicesRabbitMQ Messaging System

Rick Kohrs

Satellite Data ServicesSpace Science & Engineering Center University of Wisconsin – Madison

2018 McIDAS Users’ Group Meeting22-23 May 2018

Madison, WI

GOES-R Rebroadcast (GRB)

GroundStation Antenna Electronics Ingest System

Server

ADDE SFTP/LDMNFS

• What is RabbitMQ

• Benefits

• Satellite Data Services Paradigm

• Use Case

• What is RabbitMQ• Publisher

Publisher

Satellite IngestorADDE Server

Archive ServerProduct Generator

• What is RabbitMQ• Publisher (currently ~200 msgs/minute)

• Broker Message Queue

PublisherMessaging Queue




• Message Queue – 'The Broker'• Client/Consumer


Client




• Message Queue• Client/Consumer

• image start• band end• image complete


Client




• Message Queue• Client/Consumer

• image start• band end• image complete

• Advanced Message Queueing Protocol (AMQP)• Pika – python implementation of AMQP

Publisherpika

AMQP

Messaging Queue

pika

AMQPClient




• Exchange / Topic• satellite

• Topic Keys

sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status

Publisherpika

AMQP

Messaging Queue

pika

AMQPClient





• Topic Keys


geoleo

Publisherpika

AMQP

Messaging Queue

pika

AMQPClient





• Topic Keys


geoleo

goeshimawari

Publisherpika

AMQP

Messaging Queue

pika

AMQPClient





• Topic Keys


geoleo

goeshimawari

g16g17h8

Publisherpika

AMQP

Messaging Queue

pika

AMQPClient





• Topic Keys


geoleo

goeshimawari

g16g17h8

abiglmexis

magnetometerseisssuviahi

Publisherpika

AMQP

Messaging Queue

pika

AMQPClient





• Topic Keys


geoleo

goeshimawari

g16g17h8

abiglmexis


addefile

Publisherpika

AMQP

Messaging Queue

pika

AMQPClient





• Topic Keys


geoleo

goeshimawari

g16g17h8

abiglmexis


addefile

sdiadde

Publisherpika

AMQP

Messaging Queue

pika

AMQPClient





• Topic Keys


geoleo

goeshimawari

g16g17h8

abiglmexis


addefile

sdiadde

ncdfhsf

Publisherpika

AMQP

Messaging Queue

pika

AMQPClient





• Topic Keys


geoleo

goeshimawari

g16g17h8

abiglmexis


addefile

sdiadde

ncdfhsf

bandimage

productgranule

Publisherpika

AMQP

Messaging Queue

pika

AMQPClient





• Topic Keys


geoleo

goeshimawari

g16g17h8

abiglmexis


addefile

sdiadde

ncdfhsf

bandimage

productgranule

startend

complete

Publisherpika

AMQP

Messaging Queue

pika

AMQPClient





• Topic Keys• Body/Payload

satellite_family: GOES satellite_location: GOES-East coverage: Mesoscale-1satellite_ID: G16 start_time: 2018-04-30 19:52:29.2 mode: 3instrument: ABI end_time: 2018-04-30 19:52:35.0 signal_type: grbmedium: adde create_time: 2018-04-30 19:52:39.3 data_type: RadM1server_type: sdi server_ip: satbuf1.ssec.wisc.edu title: ABI L1b Radiancesmessage_type: band adde_dataset: EASTA/M1status: end band: 3path: /data/goes/grb/goes16/2018/2018_04_30_120/abi/L1b/RadM1/

OR_ABI-L1b-RadM1-M3C03_G16_s20181201952292_e20181201952350_c20181201952393.nc

Publisherpika

AMQP

Messaging Queue

pika

AMQPClient



• What is RabbitMQ• Client

• Exchange / Topic• Subscribe to Keys• amqpfind http://www.ssec.wisc.edu/datacenter/amqpfind/


Publisherpika

AMQP

Messaging Queue

pika

AMQPClient



geo.goes.g16.abi.adde.sdi.ncdf.band.end

• RabbitMQ Benefits• Avoid multiple data request

• cron• Immediate Access to data

• adde• https

• No need for e-mail messages

Serveradde

00

12

0618

new dataavailable

Client

wait

no



• adde• https


00

12

0618

Serveradde

new dataavailable

Client

no

wait



• adde• https


00

12

0618

Serveradde

new dataavailable

no

wait

Client

yes

processdata



• adde• https


Serveradde

00

12

0618

processdata

you've got data

Broker

Client

fanout 1

demod

GRB RF

CADU

Satellite Data ServiceGOES-16 Data Flow

fanout 1

demod

GRB RF

CADU

Satellite Data ServicesGOES-16 Data Flow

SDI

cspp-geo

SATBUF1

rsync

mq1RabbitMQ

server

publishmessages

publishmessages

rsync

arcwrite1

Lustrearcdata

fanout 1

demod

GRB RF

CADU


SDI

cspp-geo

SATBUF1

rsync

OutsideUsers

In-HouseUsers

amqpfind

https/ADDEhttps/ADDE/nfs

nfs

ADDE

mq1RabbitMQ

server

publishmessages

publishmessages

rsync

arcwrite1

Lustrearcdata

fanout 1

demod

GRB RF

CADU


SDI

cspp-geo

SATBUF1

rsync

publishmessages

OutsideUsers

In-HouseUsers

amqpfind


ADDE

nfs

mq1RabbitMQ

server

publishmessages

fanout 1

demod

GRB RF

CADU


SDIsdigrbbeta

SATBUF1

GRBGAMMA

arcwrite2

SDIsdigrbbeta

csppbeta

GEOCAT/AITLevel 2+Products

fanout 2

CADU

demod

mq2RabbitMQ

server

SATBUF2

rsyncrsync

rsync/nfs

LustreAWG

LustreUsers

OutsideUsers

In-HouseUsers

https/ADDE

ADDE

nfs

publishmessages

publishmessages

nfs

Amazon

cspp-geo dbINGE

GRBDELTA

dbINGEcspp-geo

scorescore

rsync/nfs

csppalpha


arcwrite1

Lustrearcdata

rsyncrsync

rsyncrsync

amqpfind

OutsideUsers

In-HouseUsers

amqpfind


ADDE

nfs

mq1RabbitMQ

server

publishmessages

publishmessages

rsync

arcwrite1

Lustrearcdata

rsync

fanout 1

demod

GRB RF

CADU


SDIsdigrbbeta

SATBUF1

rsync

GRBGAMMA

arcwrite2

SDIsdigrbbeta

csppbeta


fanout 2

CADU

demod

mq2RabbitMQ

server

SATBUF2

rsyncrsync

rsync/nfs

LustreAWG

LustreUsers

OutsideUsers

In-HouseUsers

https/ADDE

ADDE

nfs

publishmessages

publishmessages

nfs

NOAA Star

NOAA PDA

hwcloudtop2rsync(push)

Amazon

cspp-geo dbINGE

GRBDELTA

dbINGEcspp-geo

scorescore

rsync/nfs

csppalpha


publishmessages

NOAA Star

NOAA PDA


UNIDATA

rsync

amqpfind

OutsideUsers

In-HouseUsers

amqpfind


ADDE

nfs

mq1RabbitMQ

server

publishmessages

publishmessages

rsync

arcwrite1

Lustrearcdata

rsync

fanout 1

demod

GRB RF

CADU


SDIsdigrbbeta

SATBUF1

rsync

GRBGAMMA

arcwrite2

SDIsdigrbbeta

csppbeta


fanout 2

CADU

demod

mq2RabbitMQ

server

SATBUF2

rsyncrsync

rsync/nfs

LustreAWG

LustreUsers

OutsideUsers

In-HouseUsers

https/ADDE

ADDE

nfs

publishmessages

publishmessages

nfs

NOAA Star

NOAA PDA


Amazon

cspp-geo dbINGE

GRBDELTA

dbINGEcspp-geo

scorescore

rsync/nfs

csppalpha


publishmessages

NOAA Star

NOAA PDA


UNIDATA

rsync

amqpfind

./amqpfind -H mq2.ssec.wisc.edu -u sdsuser -p sdsmq \-X satellite \-C 'geo.goes.*.*.*.*.*.*.*' \-j '{start_time} {coverage} {instrument} {message_type} {status}'

Single Server

./amqpfind -H mq2.ssec.wisc.edu -u sdsuser -p sdsmq \-H mq1.ssec.wisc.edu -u sdsuser -p sdsmq \-X satellite \-w 60.0 \-C 'geo.goes.*.*.*.*.*.*.*' \-j '{start_time} {coverage} {instrument} {message_type} {status}'

Multi-Server

$HOME/amqpfind/amqpfind -X satellite -C 'geo.goes.*.*.*.*.*.image.complete' \-H mq1.ssec.wisc.edu -p sdsmq -u sdsuser \-j '{start_time} {server_ip} {adde_dataset}'|\grep --line-buffered -i CONUS| \xargs -P1 –n4 python $HOME/bin/mc-example.py

Wrapper Script

#! /usr/bin/pythondef main(cmdArgs):

import mcidasximport datetimeimageDate = cmdArgs[1]imageTime = cmdArgs[2]server = cmdArgs[3]addeDataset = cmdArgs[4]group = addeDataset.split('/')[0]

mcenv = mcidasx.mcenv(f=['3@750x1250'], i=228, g=8)

mccmdout = mcenv.dataloc('ADD ' + group + ' ' + server)dateTimeStr = imageDate + ' ' + imageTimedateTimeObj = datetime.datetime.strptime(dateTimeStr,'%Y-%m-%d %H:%M:%S.%f')dateTimeKeyword = dateTimeObj.strftime(' DAY=%Y%j TIME=%H:%M:%S ')mcCmd = ' USER 1234'mcCmdOut = mcenv.logon(mcCmd)mcCmd = addeDataset + dateTimeKeyword + ' BAND=13'mcCmdOut = mcenv.imgdisp(mcCmd)mcCmd = ' X current-G16-band13.gif'mcCmdOut = mcenv.frmsave(mcCmd)

if __name__ == '__main__':import sysmain(sys.argv)

mc-example.py

Thank You!

Questions?

Documents

SSEC Satellite Data Services RabbitMQ Messaging System · CADU Satellite Data Services GOES-16 Data Flow SDI sdigrbbeta SATBUF1 rsync GRBGAMMA arcwrite2 csppbeta GEOCAT/AIT Level