View
219
Download
0
Category
Preview:
Citation preview
NON-GAUSSIAN 4D VAR
STEVEN J. FLETCHER and MANAJIT SENGUPTA
With thanks to Milija and Dusanka Zupanski, Andy Jones, Laura Fowlerand Thomas Vonder Haarand Thomas Vonder Haar
DoD Center for Geosciences/Atmospheric Research (CG/AR)p ( )
COOPERATIVE INSTITUTE FOR RESEARCH IN THE ATMOSPHERE
COLORADO STATE UNIVERSITY
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 1
PRESENTATION OUTLINE
MOTIVATION FOR THE WORK
REAL LIFE EXAMPLES OF NON-GAUSSIAN VARIABLES
MATHEMATICAL ILLUSTRATIONS OF THE DRAWBACKS OF CURRENT METHODSMETHODS
PROBABILISTIC APPROACH USING CONDITIONAL INDEPENDENCE
4 D HYBRID LOGNORMAL NORMAL DATA ASSIMILATION4-D HYBRID LOGNORMAL – NORMAL DATA ASSIMILATION
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 2
MOTIVATION
An important assumption made in variational and ensemble data assimilation is that the state variables and observations are Gaussianassimilation is that the state variables and observations are Gaussian distributed
Note: The difference between two Gaussian variables is also a Gaussian variable.
Is this true for all state variables?
Is this true for all observations of the atmosphere?
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 3
REAL LIFE EXAMPLES
This data is column water vapour climatologies from the Oklahoma ARM-SGP site from 1997-2000 where the data are of when a boundary layer cloud was presentcloud was present
We have taken the observations and have sorted them by season as well as for the whole year. The data was collected using a microwave radiometer.
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 4
1
BEST LOGNORMAL AND NORMAL FITS FOR CWV FOR DJF
CWV DJFBEST LOGNORMAL FITBEST NORMAL FIT
0.6
0.8
sity
BEST LOGNORMAL AND NORMAL FITS FOR CWV FOR MAM
CWV MAM
0.4
0.6
Den
si
0.4
0.5
CWV MAMBEST LOGNORMAL FITBEST NORMAL FIT
1 2 3 4 5 60
0.2
0.3
0.4
Den
sity
1 2 3 4 5 6COLUMN WATER VAPOR
0.2
De
1 2 3 4 5 60
0.1
COLUMN WATER VAPOR
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 5
1 2 3 4 5 6COLUMN WATER VAPOR
0.35
0.4
0.45
BEST LOGNORMAL AND NORMAL FITS FOR CWV FOR JJA
0.25
0.3
0.35
ensi
ty
0.1
0.15
0.2Den
CWP JJA
BEST LOGNORMAL FIT BEST LOGNORMAL AND NORMAL FITS FOR CWV FOR SON
1.5 2 2.5 3 3.5 4 4.5 5 5.5 60
0.05
COLUMN WATER VAPOR
BEST LOGNORMAL FIT
BEST NORMAL FIT
0.35
0.4
BEST LOGNORMAL AND NORMAL FITS FOR CWV FOR SON
CWV SONBEST LOGNORMAL FITBEST NORMAL FIT
0.2
0.25
0.3en
sity
0.1
0.15
0.2Den
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 61 2 3 4 5 60
0.05
COLUMN WATER VAPOR
BEST LOGNORMAL AND NORMAL FITS FOR CWV FOR ALL SEASONS
0.4
0.45
BEST LOGNORMAL AND NORMAL FITS FOR CWV FOR ALL SEASONS
CWV ALL SEASONSBEST LOGNORMAL FITBEST NORMAL FIT
0.25
0.3
0.35
sity
0.15
0.2
0.25
Den
sit
0.05
0.1
1 2 3 4 5 60
COLUMN WATER VAPOR
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 7
Current Techniques used with non-Gaussian Variables
1) Transform by taking the LOGARITHM of the original state variable. This then makes the new variable ALMOST GAUSSIAN. Minimize the cost function with respect to this variable, TRANSFORM BACK and initialize with this state. STATE FOUND IS A NON-UNIQUE MEDIAN OF THE ORIGINAL VARIABLE, (Fletcher and Zupanski 2006a, 2007).
2) Assumed Gaussian assumption and BIAS CORRECT2) Assumed Gaussian assumption and BIAS CORRECT.
3) Using a Markov-Chain Monte-Carlo approaches (Posselt et al. 2008)
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 8
1.8PLOT OF LOGNORMAL DISTRIBUTIONS
σ=0.25
1.4
1.6σ=0.25σ=0.5σ=1σ=1.5
1
1.2
DF
=1.5
0.6
0.8PD
F
0.2
0.4
0.6
0 0.5 1 1.5 2 2.5 30
0.2
x
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 9
x
1.8PLOT OF TRANSFORMED NORMAL DISTRIBUTIONS
σ=0.25
1.4
1.6σ=0.25σ=0.5σ=1σ=1.5
1
1.2
DF
0.6
0.8PD
F
0.2
0.4
0.6
−2 −1.5 −1 −0.5 0 0.5 1 1.5 20
0.2
ln x
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 10
ln x
1.8PLOT OF LOGNORMAL DISTRIBUTIONS
1.8PLOT OF TRANSFORMED NORMAL DISTRIBUTIONS
1.4
1.6
1.8σ=0.25σ=0.5σ=1σ=1.51.4
1.6
1.8σ=0.25σ=0.5σ=1σ=1.5
0.8
1
1.2
PD
F
0.8
1
1.2
PD
F
0.2
0.4
0.6
0.2
0.4
0.6
0 0.5 1 1.5 2 2.5 30
0.2
x−2 −1.5 −1 −0.5 0 0.5 1 1.5 20
0.2
ln x
All skewness information is lost
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 11
0.2
0.22
0.24
RANDOM LOGNORMAL SAMPLE WITH μ=0, σ=0.5, SAMPLE=20,000
a) FIRST SAMPLELOGNORMAL BEST FIT 0.16
0.18
RANDOM LOGNORMAL SAMPLE WITH μ=0, σ=1, SAMPLE=20,000
b) SECOND SAMPLELOGNORMAL BEST FIT
0.1
0.12
0.14
0.16
0.18
0.2
PD
F
0.08
0.1
0.12
0.14
PD
F
0.02
0.04
0.06
0.08
0.1
0.02
0.04
0.06
0.08
0 1 2 3 4 5 60
0.02
X0 1 2 3 4 5 6 7 8 9 10
0
X
0.12DIFFERENCE BETWEEM THE TWO LOGNORMAL SAMPLES
d)DIFFERENCES
0.06
0.08
0.1
DF
DIFFERENCES
NOTE: DIFFERNCE IS NOT
ASSUMED GAUSSIAN APPROACH
0.02
0.04
0.06
PD
A GAUSSIAN
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 12
−10 −8 −6 −4 −2 0 2 40
X
0.2
0.22
0.24RANDOM LOGNORMAL SAMPLE WITH μ=0, σ=0.5, SAMPLE=20,000
a) FIRST SAMPLELOGNORMAL BEST FIT
0.07
0.08
0.09RANDOM LOGNORMAL SAMPLE WITH μ=1, σ=0.5, SAMPLE=20,000
b) SECOND SAMPLELOGNORMAL BEST FIT
0.1
0.12
0.14
0.16
0.18
PD
F
0.04
0.05
0.06
0.07
PD
F
0.02
0.04
0.06
0.08
0.1
0.01
0.02
0.03
0 1 2 3 4 5 60
0.02
X
0 1 2 3 4 5 6 7 8 9 100
X
0.07
0.08DIFFERENCE BETWEEM THE TWO LOGNORMAL SAMPLES
d) DIFFERENCES
0.04
0.05
0.06
0.07
DF
NOTE: DIFFERNCE IS NOT
0.02
0.03
0.04
PD
F
A GAUSSIAN
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 13−10 −9 −8 −7 −6 −5 −4 −3 −2 −1 0 1 2 3 4 50
0.01
X
PROBLEMS ASSOCIATED WITH CURRENT TECHNIQUESQ
ASSUMED GAUSSIAN:
IMPACT 1: Wrong probabilities assigned to outliers.
IMPACT 2: Probabilities assigned to unphysical values.
IMPACT 3: Wrong statistic used to approximate the true distribution of the random variable.
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 14
MISCONCEPTIONS ABOUT LOGNORMAL DATAMISCONCEPTIONS ABOUT LOGNORMAL DATA ASSIMILATION
1) The theor holds as the backgro nd sol tion is independent of the tr e1) The theory holds as the background solution is independent of the true solution, it is only an approximation and statistically has no information about the true solution.
2) The theory holds for the observational component as the observations are independent of the observations operator and vice-versa.
3) If two solutions have a relative error of 50% then we are still out by a factor of two in both cases no matter what order of magnitude.
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 15
4D LOGNORMAL DATA ASSIMILATIONUnlike with the three dimensional version of variational data assimilation, the four dimensional version is defined as a weighted least squares problem.
Th G i i ht d l t h t 4D VARThe Gaussian weighted least squares approach to 4D VAR is defined through a calculus of variation problem with initial conditions found through the adjoint.
This weighted least squares approach can be defined for aThis weighted least squares approach can be defined for a lognormal framework, which is defined by the following inner product
N
( ) ( )( ) ( )( )( )∫∫∫∑=
− −−=A
N
iiiiiiii
o
g1
01
01 lnln,lnln21 xMhyRxMhyx0
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 16
As with the Gaussian case we know that the first variation of the functionalAs with the Gaussian case we know that the first variation of the functional defined on the previous slide is equivalent to
0N
( ) ( )( )1
01
,001 ,lnln0
xRHWxMhyx δδgN
iiiiioiii −=∑
=
−M
( ) 001
1
, xx δgi
∇==
Through using the properties of inner products we get that the gradient is
( ) ( ) ( )( )( )1N
To
∑( ) ( ) ( )( )( )01
1,1 lnln xMhyRHWx0 iii
i
Tiiiiog −=∇ ∑
=
−M
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 17
The solution is a median and not the mode and hence is independent of the variance.
We need to define the functional as
( ) =g x( )
( )( ) ( )( )( )∫∫∫∑ − −+−
=N
iiiiT
iii
o
g
01
0
2
lnln,lnln21 xMhyR1RxMhy
x0
( )( ) ( )( )( )∫∫∫∑=A i
iiiiiii1
00 ,2
yy
Which then has a gradient of
( ) ( ) ( )( )( )1RxMhyRHWx T+∇ ∑ −1 lnlnN
To
g M
Which then has a gradient of
( ) ( ) ( )( )( )1RxMhyRHWx i0 +−=∇ ∑=
01
,2 lnln iiii
iiiiog M
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 18
Current Gaussian approachN
I d T f t h i
( ) ( )( )( ) ( )( )( )∑=
− −−=0
10,00,0
1,0 ,
N
iiiiiiiiGG MMJ xhyxhyRx
Improved Transform technique
( ) ( )( )( )( ) ( )( )( )( )∑ − −−=0
0,00,01,0 lnln,lnln
N
iiiiiiiLTR MMJ xhyxhyRx
New Lognormal 4D VAR approach:
( ) ( )( )( )( ) ( )( )( )( )∑=1
,,,i
( ) ( )( )( )( ) ( )( )( )( )∑=
− −+−=0
1,01
0,0,0,01,0 lnln,lnln
N
iiiiNiLiiiiLLN MMJ xhy1RxhyRx
Term that gives the mode
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 19
( )N321N210 y,,y,y,yx,,x,x,x0
KK =Po
( ) ( ) ( ) ( )( )
0112011010 x,xyxx,xyxxx0
PPPPo
,
( )011N1NNN x,x,y,,x,y,xyooKK −oP
The expression above is Bayes theorem for a multi-event probability situation. It can be simplified through using conditional independence.
( ) ( ) ( )∏=oN
PPP xyxyyyyxxxx( ) ( ) ( )∏=
=o
iiPPP
100N321N210 xyxy,,y,y,yx,,x,x,x
0KK
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 21
Taking the negative logarithm of the circled pdf in the previousTaking the negative logarithm of the circled pdf in the previous slide results in
⎫⎧ N
( ) ( ) ( )⎭⎬⎫
⎩⎨⎧
−−= ∑=
0
1000 lnlnmin
N
iiPPJ xyxx
This can now be used to derive a 4D VAR system for any distributed random variable
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 22
( )( )( )
=
∏
N321N210 y,,y,y,yx,,x,x,x0
N
P
o
oKK
( ) ( )=∏ 00 xyxi
iPP1
hG il i iF h
( ) ( ) ( )⎭⎬⎫
⎩⎨⎧ −−−∝ −
b 00b 000 xxBxxx TP 10
1exp
havewecaseGaussian temultivaria For the
( ) ( ) ( )
( ) ( )( )( ) ( )( )( )⎬⎫⎨⎧ −−−∝
⎭⎬
⎩⎨∝
−
b,00b,000
xMhyRxMhyxy
xxBxxx
TP
P
1
0
1exp
2exp
( ) ( )( )( ) ( )( )( )⎭⎬
⎩⎨ −−−∝ 0i0i0 xMhyRxMhyxy iiiiiiP
2exp
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 23
( ) ⎟⎞
⎜⎛∏ i0,x
N
P
havewecaselognormaltemultivariaFor the
( )
( ) ( )⎫⎧
×⎟⎟⎠
⎜⎜⎝
∝=∏
jb,0,
i0,0 xx
T
j
P
1
1
1 ( ) ( )
( )( )⎟⎞⎜⎛
⎭⎬⎫
⎩⎨⎧ −−− −
b,00b,00
xM
xxBxx
N
T
io h
10 lnlnlnln
21exp
,
( ) ( )( )
⎫⎧
×⎟⎟⎠
⎞⎜⎜⎝
⎛∝
=∏0
xMxy
k ki
kii y
hP
1 ,
0,,
( )( )( ) ( )( )( )⎭⎬⎫
⎩⎨⎧ −−− −
0i0i xMhyRxMhy iiiT
ii1
21exp
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 24
Results with the Lorenz 1963 modelResults with the Lorenz 1963 model
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 25
We are going to be using the hybrid Gaussian-lognormal distribution andWe are going to be using the hybrid Gaussian lognormal distribution and comparing it with the transform approach. The associated cost function is
( ) ( ) ⎞⎛T1 0( ) ( ) ( ) ⎟⎠
⎞⎜⎝
⎛+−−= −
NN
qb
pbTbb
Tb
Tb
TbJ
,
,102
110
εεεBεεx
( ) ( ) ⎟⎠
⎞⎜⎝
⎛+−−+ ∑∑
==
−
iqo
ipoN
i
Tioio
Tio
N
ii
Tio
Tio
oo
,,
,,
1,,,
1
1,, 1
0εεεRεε
ii 11
( )( )⎟
⎞⎜⎛ −
=⎟⎟⎞
⎜⎜⎛ −
= hxhy
εxx
ε ipipi
pbpb
,,,llll
1
( )⎟⎠⎜⎝ −⎟⎟
⎠⎜⎜⎝ − xhyεxxε o
iqiqi
qbqb
,,,
, lnlnlnln1
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 26
Example with the Lorenz 1963 model
The three non-linear differential equations are given by (Lorenz 1963)
yxx σσ +−=&28AND108β
zxyzyxxzy
y
βρ
=−+−=
&
&28AND10,
3=== ρσβ
zxyz β−=
5606.22 AND 4841.5,4458.5 000 =−=−= zyx
Going to assume x and y components and the associated obs are Gaussian z is lognormalGaussian, z is lognormal
(Fletcher and Zupanski 2007)
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 27
Plots of the differences in the trajectories with manytrajectories with many
accurate obs with short assimilation windows
Note transform andNote: transform and hybrid approach
quite similar
BEST CASE SCENERIO
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 28
SCENERIO
When assimilation window is l (1000t ) ith f dlarge (1000ts) with fewer and
less accurate obs, hybrid approach is more accurate
Transform approach converges quickly toconverges quickly to the wrong solution
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 29
Conclusions and Further WorkCareful which statistic to use to analyses yMode is closer to the true trajectory in the Lorenz 63 modelPossible to assimilate variables of mixed types simultaneously Combine other distributions?? i.e. Gamma, Normal, LognormalMore consistent methods for finding positive definite variablesN d t h b k d i t i (if l dNo need to change background error covariance matrix (if already
using the transform approach)
CSU/CIRA Steven J. Fletcher 8th Adjoint Workshop, May 21st 2009 30
Recommended