Use of Satellite Measurements in Data Assimilationsharaku.eorc.jaxa.jp/IPWG7/handout/...Boukabara.pdf · Use of Satellite Measurements in Data Assimilation: ... Core Mathematical

Use of Satellite Measurements in Data Assimilation:

Overview of NOAA efforts to improve the assimilation of satellite data in support of US Joint Center for Satellite Data Assimiation (JCSDA)

S-A. Boukabara

The IPWG7 training course program , 17-20 November 2014, Tsukuba, Japan

NOAA/NESDIS/STAR & JCSDA

2

Contents

Introduction & Concepts 1

Why users should be interested in Data Assimilation? 2

Why are satellites important for Assimilation & Forecasts? 3

What is Involved in Satellite Data Assimilation? 5

Different Sensors for Different Applications 4

Conclusions & Look into the future 6

3

JCSDA

JCSDA

NASA GSFC

NOAA NWS

U.S. Air Force

NOAA OAR

NOAA NESDIS

U.S. Navy

Mission:…to accelerate and improve the quantitative use of research and operational satellite data in weather,

ocean, climate and environmental analysis and prediction models.

Vision: An interagency partnership working to

become a world leader in applying satellite data and research to operational goals in

environmental analysis and prediction

Satellite Data Assimilation

4

Data Assimilation (Provides Initial

Conditions) Satellite Data

Background (from Fcst)

Current Operational Global Forecasts

Non-Satellite Data

Regional Forecasts

Other Forecasts

5

What do satellites measure? All-Weather Radiative Transfer

Scattering Effect

Scattering Effect

Absorption

Surface

sensor Satellite Data Assimilation is therefore able to analyze: -Atmosphere (Temperature, moisture, aerosols, …) -Surface (ice, snow, land, ocean) -Hydrometeors (cloud, rain, suspended ice)

Sensors Types

vDepending on targeted phenomena, sensors would be (in order of importance for DA) § Microwave sounders (sounding of T, Q) § Hyperspectral Infrared sensors (sounding, trace

gases, etc) § Radio Occultation sensors (temperature sounding) § Infrared Sensors from Geo platforms (AMVs, ..) § Active sensors (wind, wave height, etc) § Microwave Imagers (Precipitation, cloud, …) § Etc

6

Satellite Data Assimilation

7

Forward Operator (such as CRTM)

X: State vector of Geophysical

Parameters (T, Q, Tskin, etc)

Y: State vector of Radiometric Measurements (MW, IR, etc)

K: Jacobians dY/dX

Data Assimilation

X (analysis) Y + K

Non-satellite Data

8

Concept Diagram of Satellite Data Assimilation

Raw Measurements Level 1B Tbs

Radiance Processing & Radiometric Bias

Monitoring

Analyses

Background

Data Assimilation

Radiometric Bias Ready-To-Invert

Radiances

RTM Uncert. Matrx F

NEDT Matrx E

Thinning

9

Core Mathematical Basis

Main Goal in ANY Retrieval System is to find a vector X with a maximum probability of being the source

responsible for the measurements vector Ym

Main Goal in ANY Retrieval System is to find a vector X: P(X|Ym) is Max

In plain words:

Mathematically:

P(Y)Y)|P(XP(X)X)|P(YY)P(X, ´=´=

Bayes Theorem (of Joint probabilities)

)mP(YP(X)X)|mP(Y)mY|P(X ´=

=1

10

Mathematically:

Core Mathematical Basis

P(X)X)|mP(Y ´

Main Goal in ANY Retrieval System is to find a vector X with a maximum probability of being the source

responsible for the measurements vector Ym

Main Goal in ANY Retrieval System is to find a vector X: P(X|Ym) is Max

In plain words:

Problem reduces to how to maximize:

Probability PDF Assumed Gaussian around Background X0 with a

Covariance B

úúú

û

ù

êêê

ë

é÷ø

öçè

æ÷ø

öçè

æ -´-´-- 0XX1B

T0XX2

1exp

Mathematically: Probability PDF Assumed Gaussian around Background Y(X) with a

Covariance E

úúú

û

ù

êêê

ë

é÷øö

çèæ

÷øö

çèæ -´-´-- Y( X)mY1ETY( X)mY21exp

ï þ

ï ý

ü

ï î

ï í

ì

ú ú ú

û

ù

ê ê ê

ë

é ÷ ø ö

ç è æ

÷ ø ö

ç è æ

ú ú ú

û

ù

ê ê ê

ë

é ÷ ø ö

ç è æ

÷ ø ö

ç è æ - ´ - ´ - - ´ - ´ - ´ - - Y(X) m Y 1 E

T Y(X) m Y 2 1 exp 0 X X 1 B

T 0 X X 2

1 exp

Maximizing

ï þ

ï ý ü

ï î

ï í ì

ú ú ú û

ù

ê ê ê ë

é ÷ ø ö ç è

æ ÷ ø ö ç è

æ ú ú ú

û

ù

ê ê ê

ë

é ÷ ø ö ç

è æ

÷ ø ö ç

è æ - ´ - ´ - - ´ - ´ - ´ - - Y(X) m Y 1 E T Y(X) m Y 2

1 exp 0 X X 1 B T 0 X X 2

1 exp

Is Equivalent to Minimizing

÷øöç

èæ- )mY|P(Xln

Which amounts to Minimizing J(X) –also called COST FUNCTION – Same cost Function used in 1DVAR Data Assimilation System

( ) ( ) ( ) ( )úûù

êëé -´´-+úû

ùêëé -´´-= -- Y(X)YEY(X)Y

21XXBXX

21J(X) m1Tm

01T

0

=)mY|P(X

11

vTo find the optimal solution, solve for: vAssuming Linearity

vThis leads to iterative solution:

Cost Function Minimization

0(X)'JXJ(X) ==¶

¶

úûù

êëé

÷øöç

èæ

ïïþ

ïïý

ü

ïïî

ïïí

ì

÷÷ø

öççè

æ +----+-=+ nΔXnK)nY(XmY1ETnK1

nK1ETnK1B1nΔX

úûù

êëé

÷øöç

èæ

ïïþ

ïïý

ü

ïïî

ïïí

ì

÷÷ø

öççè

æ +--

+=+ nΔXnK)nY(XmY1

ETnBKnKTnBK1nΔX

úúû

ù

êêë

é -+= 0xxK)0y(xy(x)

More efficient (1 inversion) Preferred when nChan << nParams (MW)

Preferred when nParams << nChan (IR)

12

Variational Retrieval/Assimilation Measured Radiances

Initi

al S

tate

Vec

tor

Solution Reached

Forward Operator

Simulated Radiances Comparison: Fit

Within Noise Level ?

Update State Vector

New State Vector

Yes

No Jacobians

Geophysical Covariance

Matrix B

Measurement & RTM

Uncertainty Matrix E

Geophysical Mean

Background

Climatology (Retrieval Mode) Forecast Field (1D-Assimilation Mode)

13

Most Important Assumptions Made in Solution Derivation

vLocal linearity vGaussian Distribution of the Instrument/Model

Errors vGaussian Distribution of the Background Error vIndependence between Instrument/Model

errors and Background Errors

14

Multi-Dimensional Problem Leads to Higher chances of Local Minima

J(X)

X stGX 1

1stGX 1

2optX

In this case, it does not matter what 1st guess is used

J(X)

X stGX 1

1stGX 1

2optX

In this case, 1st guess is crucial

Importance of First Guess Background

15

Contents



Why are satellites important for Assimilation & Forecasts? 3




Future Merged SA and NG Product Generation “Provide what the forecasters need, when they need it…”

SDRs (Polar)

Metop, N19, NPP, DMSP

IR, MW

SDRs (Geo)

GOES, GOES-R,

MSG,

Ground-Baaed Data

Radar Conventiona

l Data

Airborne Data

GPS Data

Environment Analysis –Geophysical products

(Data Fusion)

Common Data Assimilation & Data Fusion Tool

- Combine DA and RS Expertise - Highly flexible to serve as

- Platform for O2R/R2O - Complete Analysis (atmosphere,

cryosphere, ocean, land, hydrometeors, etc)

“SA”

AWIPS


(Data Fusion)

NG Mode (In NWS):

- Closely tied to Forecast Model, - Every 6 hours

“NG” SA Mode (In

NESDIS): - Data Fusion of

all sensors, -Every hour

Forecaster

16

17

Contents



Why are satellites important for Assimilation/Forecasts 3




18

OSE Activities / Impact Assessment

– An extensive assessment of the global observing system impact on NOAA forecast system has been undertaken.

– The impact assessment was

done wrt satellite data (collectively &individually: microwave AMSU, MHS, GPS, hyperspectral IR, AMVs, etc) as well as conventional data.

– Satellite data as a group, had a very significant impact which surpasses the conventional data impact (by a wide margin), especially in the southern hemisphere.

Results from the extensive data denials experiments performed in the JCSDA, aimed at assessing the impact of the global Plots courtesy of J. Jung.

19 Sun

N-15(PM)

N-16(AM)

N18(PM)

METOP-A(AM)

N17(AM)

Constellation as of September 2012. Sources: NESDIS/OSO & CGMS/WMO pages

16:41

Mean Local Times at the Ascending Node (hh:mm)

20:25

19:17

14:48

21:30

12:00 Noon

N-19(PM)

13:32

NPP(PM)

13:30

00:00

18:00 06:00

18:50

F16(Early AM) 17:37

F17(Early AM)

20:08

F18(AM)

METOP-B(AM)

21:40

F19(Early AM) 17:xx

Aqua (PM)

13:30

Future satellite (high level confidence)

19:xx

F20(AM)

Impact Assessment of the Orbital Distribution

The “Control” experiment establishes baseline performance of the current satellite observing system capability.

The “3polar” experiment removes the redundancy of observations but maintains platforms flying in early-morning, mid-morning, and afternooon polar orbits.

The “2polar” experiment removes the redundancy of observations plus all platforms flying in the afternoon polar orbit (the data gap).

The “1polar” experiment removes the redundancy of observations plus both the early-morning and afternoon polar orbits (leaving only mid-morning observations).

Importance of Orbital Coverage

20

21

Contents



Why are satellites important for Assimilation/Forecasts? 3




AMSU and MHS v Main purpose of the

microwave sensors (AMSU and MHS) is the atmospheric sounding (T and WV)

v AMSU has two modules (A-1 and A-2) with channels ranging from 23.8 GHz to 89 GHz

v AMSU has 30 scan positions per scanline

v MHS probes at frequencies between 89 and 183 GHz

v MHS has a higher spatial resolution (90 scan positions per scanline)

23

AMSR-2 Data Assimilation

AMSR2 Soil Moisture (Reference)

GFS-EnKF: After AMSR2 Soil Moisture DA

GFS: Without AMSR2 Soil Moisture DA

Assimilation of

Satellite Soil Moisture Product from AMSR2 in Global Forecast System.

In this figure, the

Noah LSM multiple year means and standard deviations are used to scale the surface layer soil moisture retrievals before

OSCAT Data Assimilation

vOSCAT DA was improved in the next version of the GDAS system vPre-assessment of data is

necessary to optimize and characterize filtering, thinning, biases and observation error estimates v Errors in wind direction has

a bigger effect on A/C than intensity vOSCAT has since died

24

Impact assessment of the OSCAT scatterometer data assimilation. These plots represent the forecast impact (b) and verification results (a) of OSCAT winds experiments. They represent the change in anomaly correlation and RMS (increase or decrease) of the surface wind speed at 0.995 sigma level. The impact, globally, at 48 hours lead time is mixed, but overall positive. Plot courtesy of Li Bi, Riverside Inc, JCSDA Active Sensors data assimilation scientist.

a

b

Assimilation of Atmospheric Motion Vector (AMV)

v Improvement of Wind analysis by optimizing the AMVs assimilation (from GOES sensors)

v U-wind component of the GDAS analysis is improved with respect to ECMWF analysis in the region of the Tropical Easterly Jet (TEJ).

26

v AC scores (the higher the better) as a function of the forecast day for the 500 mb gph in Southern Hemisphere

v 40-day experiments: § expx (NO COSMIC) § cnt (old RO assimilation

code - with COSMIC) § exp (updated RO

assimilation code - with COSMIC)

COSMIC provides 8 hours of gain in model forecast skill starting at day 4

GPS RO Assimilation

Plots courtesy of L. Cucurull. Internal JCSDA project.

27

Contents







Steps to transition a new sensor to operational data assimilation (1/2)

v 1- Pre-Launch phase: § Forward model preparation § Covariance matrix preparation for all parameters in the assimilation control vector § Preparation of the tools for preprocessing data, QC/flagging the data. § Assess hardware and software requirements needed and plan accordingly § Assess footprint averaging, thinning methodologies that are appropriate, for the

specific sensor. Thinning is understood as spatially and spectrally. § Obtain sample data § Obtain/generate decoding codes for the sample data § Initial estimates of instrument noise from data § Simulation of a full data assimilation exercise on simulated data (sample files) § Generation of a flow of simulated data, based on proxy real data (for example

ATMS data based on AMSU data) with identical format as the expected real data § Set up the ingest system, assess potential bottlenecks and fix issues. Goal:

simulate as much as possible the expected configuration after launch, before the launch.

Steps to transition a new sensor to operational data assimilation (2/2)

v 2- Post-Launch Phase: § Monitor telemetry, noise NeDT, stability of gains, hot loads, cold loads,

and other major parameters. § Assess quality of measurements (by comparing to simulation based on

forecast/analysis fields) (OB-BK) § Assess geo-location quality of measurements § Assess QC/pre-processing tools (rain flag ice flag, convergence metric,

etc) § Determine/monitor bias of measurements as well as RTM uncertainties § Test/Adjust footprint matching & thinning methodology § Perform parallel assimilation tests to determine impact on forecast skills § Full operational implementation if tests positive. § Continuous improvement/fine tuning of assimilation methodology, QC,

bias adjustment, thinning, etc.

30

Satellite Data Thinning vObjective of CSTROT:

ØDevelop a new thinning scheme to optimize satellite data usage

in GSI data assimilation for both global and regional

modeling systems. vCSTROT Functions: Ø Thinning options: • using Standard Deviation

• using regression • by skipping points

ØRepresentation options: • Random points • Closest point • Averaging

ØNested domain options: • by target regions • by domain size

CSTROT is an “intelligent” thinning tool to optimize satellite data selection in DA.

Thinning of AMSU-A (N15+N19+Metop-A) Ch-2 Tb ( 0006 UTC 23 July, 2013 )

Two domain areas

Two target regions

Higher density in higher variation regions associated with cloudy,

frontal system, moisture tongue.

Specified Auto detected The tool will allow an optimal information content

extraction while optimizing computation time

+ +

+ +

+

+

+ + +

Footprint Matching (Case of AMSU/MHS)

vFootprint matching is very sensor-dependent vDifferent Approaches

for Footprint matching: § Simple averaging § Backus Gilbert

Let’s not forget about HPC

v Data assimilation is computer- intensive

v Supercomputers are thereforw very important for satellite data assimilation

32

33

Challenges in Satellite Data Assimilation (From JCSDA)

vDifficult to ingest all satellite data due to a lack of computational resources and fast radiative transfer schemes

vDifficult to use satellite measurements that are

affected by surface vDifficult to assimilate satellite radiances that are

affected by aerosols and clouds vThere is a lot of work to be done still..

34

Contents







Current Use of Satellite Data (Numerical Guidance and Situation Awareness)

SDRs (Polar)


IR, MW

SDRs (Geo)

GOES, GOES-R,

MSG,

Algor Algor

Ground-Baaed Data

Radar Conventiona

l Data

Airborne Data

GPS Data

Prod

ucts

/Der

ived

Analysis

Data Assimilation + Internal Inv. Algos

Forecast Background

Only for “NG” AWIPS (for short term forecast)

Forecast

35

Future Merged SA and NG Product Generation “Provide what the forecasters need, when they need it…”

SDRs (Polar)


IR, MW

SDRs (Geo)

GOES, GOES-R,

MSG,

Ground-Baaed Data

Radar Conventiona

l Data

Airborne Data

GPS Data


(Data Fusion)

Common Data Assimilation & Data Fusion Tool

- Combine DA and RS Expertise - Highly flexible to serve as

- Platform for O2R/R2O - Complete Analysis (atmosphere,

cryosphere, ocean, land, hydrometeors, etc)

“SA”

AWIPS


(Data Fusion)

NG Mode (In NWS):

- Closely tied to Forecast Model, - Every 6 hours

“NG” SA Mode (In

NESDIS): - Data Fusion of

all sensors, -Every hour

Forecaster

36

Increase Role of Assimilation in NWP and Climate Reanalysis (1/2)

v Satellite data is critically used in data assimilation system (traditionally for NWP)

v Satellite Measurements by nature are sensitive to all sorts of parameters (products). If it can retrieved, it can be ‘analyzed’.

v Efforts are ongoing to extend/improve the assimilation to all sensors (active/passive, RO, IR/MW, Lightning, etc), all situations (cloudy, rainy, ice-covered, ..)

v Coupled data assimilation is becoming a major focus: this will lead to using data assimilation analysis beyond NWP (to ocean, land, cryosphere, hydrometeors, etc)

v Increase in spatial resolution (~13kms) and temporal resolution (hourly analysis are becoming more and more common) will lead to usage of assimilation for situational awareness purposes (nowcasting and short term forecasting)

vSatellite data is more and more frequently used in conjunction (or blended): MW and IR TPW, ground-based and satellite, Geo and Leo

vData assimilation presents an excellent tool to perform this data ‘fusion’ of sort.

vData assimilation is becoming the ‘entry point’ for the usage of satellite data for diverse types of users

vCross-sensors intercalibration happens naturally inside the data assimilation: A unique analysis is produced out of the hundreds of measurements types.

vThis lesser sensitivity to calibration errors is naturally extending the capability to climate applications (re-analysis)

Increase Role of Assimilation in NWP and Climate Reanalysis (2/2)

JCSDA Activities in Training, Education & Outreach

39

vMonthly Seminar Series on DA: remote access available v Summer colloquium in satellite data assimilation (3-year

cycle). Next one planned for summer 2015. Open for everyone v JCSDA Annual symposium co-organized during the next

AMS annual meeting in 215. v Annual JCSDA workshop on satellite data assimilation. Next

one planned for May 2015) v Joint Workshops with Other Programs and International

Partners. DTC-JCSDA joint workshop/tutorial, ECMWF-JCSDA workshop, etc

v JCSDA Newsletters (quarterly) v Highlight achievements by JCSDA scientists (internal/external) v Disseminate results and promote collaboration

vActive web site: jcsda.noaa.gov

Questions?

41

BACKUP SLIDES

42

Four Dimension Variational Analysis (4DVAR)

vWithin an assimilation window, recent measurements are accounted for to reduce the time-dependent cost-function and produce a new trajectory for subsequent forecast.

v Difficulties: § Adjoint in temporal

domain can be non-linear § Huge computational

requirements and storage

43

4DVAR GDAS

Above figures compare GDAS analysis temperature fields near 250 hPa and surface with 1DVAR retrievals and 4DVAR analysis. The temperature field from analysis shows hurricane warm core is about 2 degree warmer than GDAS analysis. Uses of cloudy radiances under storm conditions dramatically improve warm core structure. At 0600 UTC August 25, 2005, Katrina was at tropical storm intensity, with the minimum central pressure of 1000 hPa.

1DVAR+4DVAR: Katrina Analysis

250 hPa

Surface

Documents

Use of Satellite Measurements in Data Assimilationsharaku.eorc.jaxa.jp/IPWG7/handout/...Boukabara.pdf · Use of Satellite Measurements in Data Assimilation: ... Core Mathematical