Bayesian Gaussian

Embed Size (px)

Citation preview

  • 8/12/2019 Bayesian Gaussian

    1/22

    Multivariate Normal Distribution

    Multivariate Normal Distribution

    Brunero LiseoSapienza Universita di Roma,

    [email protected]

    February 10, 2014

    1/22

    http://find/
  • 8/12/2019 Bayesian Gaussian

    2/22

    Multivariate Normal Distribution

    Outline

    2/22

    http://find/
  • 8/12/2019 Bayesian Gaussian

    3/22

    Multivariate Normal Distribution

    Inference on N ( , )

    LetX 1, . . . , X n

    iid N p( , ).

    with density

    f x (x ; , ) = 1

    (2) p/ 2 | |1/ 2 exp

    12

    (x ) 1(x )

    Likelihood is

    L( , ) 1| |n/ 2

    exp 12

    n

    i=1(x i )

    1(x i )

    3/22

    http://find/
  • 8/12/2019 Bayesian Gaussian

    4/22

    Multivariate Normal Distribution

    Alternative expression of quadratic form

    n

    i=1(x i )

    1(x i )

    =

    n

    i=1 (x

    i x

    ) 1

    (x

    i x

    )

    =n

    i=1(x i x )

    1(x i x ) + n( x ) 1(x )

    = tr 1n

    i=1(x i x )( x i x ) + n(x ) 1( x )

    = tr 1S + n( x ) 1( x )

    Then4/22

    http://find/
  • 8/12/2019 Bayesian Gaussian

    5/22

    Multivariate Normal Distribution

    Conjugate prior

    | N p( , c 1 ),

    that is

    ( | ) 1| |1/ 2

    exp c2

    ( ) 1( )

    IW p( , ),

    then has an Inverse Wishart distribution,

    ( ) 1

    | | ( p +1)

    2

    exp 12

    tr 1 1

    5/22

    http://find/
  • 8/12/2019 Bayesian Gaussian

    6/22

    Multivariate Normal Distribution

    Wishart distribution

    (Wk(m, )) has support the space of all positive denitesymmetric matrices.We say that the square k-dimensional matrix V , positive denite,has Wishart distribution with m dof and scale parameter ,positive

    denite matrix, and we denote it by W k(m, ), if the density is

    f (V ) = 1

    2mk/ 2k (m/ 2)| |m/ 2|V | (m k 1) / 2 exp

    12

    tr 1V ,

    with

    k(u) = k(k 1) / 4

    k

    i=1 u

    12

    (i 1) , u > k 1

    2 .

    6/22

    M l i i N l Di ib i

    http://find/
  • 8/12/2019 Bayesian Gaussian

    7/22

    Multivariate Normal Distribution

    Construction of a random Wishart matrix

    Let (Z 1, , Z m ) iid N k (0 , I ); then the quantity

    W =m

    i=1

    Z i Z i

    has a Wishart distribution W k(m, I ).

    The diagonal element of W, say W jj follows a 2m distribution.

    Starting from (Z 1, , Z m ) iid N k (0 , ) we can obtain a more

    general W .

    7/22

    M lti i t N l Di t ib ti

    http://find/
  • 8/12/2019 Bayesian Gaussian

    8/22

    Multivariate Normal Distribution

    Inverse Wishart

    An Inverse Wishart r.v. (W 1k (m, ), (say IW) has support the

    space of all symmetric positive denite matrices.A IW r.v. describes the the distribution of the inverse of aWishart matrix.In Bayesian statistics it is often used as the conjugate prior forthe covariance matrix of a multivariate Gaussian model.

    8/22

    Multivariate Normal Distribution

    http://find/
  • 8/12/2019 Bayesian Gaussian

    9/22

    Multivariate Normal Distribution

    Let V W k(m, ) . Since V is pos. def. with prob. 1, it is easyto compute the density function of Z = V 1:

    f (Z ) = |Z | (m + k+1) / 2

    2mk/ 2k(m/ 2)| |m/ 2 exp 12 tr

    1Z

    1.

    Also

    E ( Z ) = 1

    m k 1.

    9/22

    Multivariate Normal Distribution

    http://find/http://goback/
  • 8/12/2019 Bayesian Gaussian

    10/22

    Multivariate Normal Distribution

    A useful Lemma

    Lemma .Let A and B be positive real numbers and let a and b be any real

    numbers. Then

    A( a)2 + B ( b)2 = ( A + B ) aA + bB

    A + B

    2

    + ABA + B

    (a b)2

    (1)

    Proof; see later

    10/22

    Multivariate Normal Distribution

    http://find/http://goback/
  • 8/12/2019 Bayesian Gaussian

    11/22

    Multivariate Normal Distribution

    (Multivariate version)

    Let x , a , b vectors in R k and let A , B be symmetric matrices k ks.t. (A + B ) 1 exists. Then,

    (x

    a

    )A

    (x

    a

    ) + (x

    b

    )B

    (x

    b

    )

    = ( x c ) (A + B )( x c ) + ( a b ) A (A + B ) 1B (a b )

    wherec = ( A + B ) 1(Aa + Bb )

    When x R the result is exactly (1)

    11/22

    Multivariate Normal Distribution

    http://find/
  • 8/12/2019 Bayesian Gaussian

    12/22

    Multivariate Normal Distribution

    Proof

    (x a ) A (x a ) + ( x b ) B (x b )

    = x (A + B )x 2x (Aa + Bb ) + a Aa + b Bb

    Add and remove c (A + B )c ,

    (x c ) (A + B ) (x c ) + G ,

    where G

    =a Aa

    +b Bb

    c

    (A

    +B

    )c

    . Alsoc (A + B )c = ( Aa + Bb ) (A + B ) 1 (Aa + Bb ) =

    (add and remove Ab in the rst and third factors)

    12/22

    Multivariate Normal Distribution

    http://find/http://goback/
  • 8/12/2019 Bayesian Gaussian

    13/22

    [A (a b ) + ( A + B )b ] (A + B ) 1 [(A + B )a B (a b )]

    = (a b ) A (A + B ) 1B (a b )+( a b ) Aa + b (A + B )a b B (a b

    (a b ) A (A + B ) 1B (a b ) + a Aa + b Bb ;

    Therefore

    G = ( a b ) A (A + B )

    1B (a b ) .

    13/22

    Multivariate Normal Distribution

    http://find/http://goback/
  • 8/12/2019 Bayesian Gaussian

    14/22

    Dickeys TheoremTheorem .Let X be a k-dimensional random vector and Y be a scalar r.v.such that

    X | Y N k ( , Y ), Y GI (a, b);

    Then the marginal distribution of X is Multivariate Student

    X St k 2a, , ba

    .

    In particular, setting a = / 2 e b = 1 / 2, then

    Y 1 2 ; X St k (, , / ).

    Proof: Easy.14/22

    Multivariate Normal Distribution

    http://find/
  • 8/12/2019 Bayesian Gaussian

    15/22

    The Posterior

    Using the Lemma, one gets | , x N p(

    , (c + n) 1 ), with

    = c + n x

    c + n

    and |x IW p( + n, ), where

    = S +

    1 + ncn + c( x ) ( x )

    1

    15/22

    Multivariate Normal Distribution

    http://find/
  • 8/12/2019 Bayesian Gaussian

    16/22

    The hyperparameters

    We need to specify the following parameters: , the prior mean for , the most reasonaable estimate beforethe experiment;

    c; the degree of believe in your elicitation about ; smallervalues of c makes the prior less informative; and m represent the hyper-parameters about 1; theycan be elicitated by taking into account the moments of anInverse WIshart inversa: for example,

    E ( ) = 1

    p m 1

    16/22

    Multivariate Normal Distribution

    http://find/
  • 8/12/2019 Bayesian Gaussian

    17/22

    Non informative case

    You get a noninformative prior if you set the Hyper-parametersequal to zero

    Whenc 0, 1 = 0 , = 0 ,

    you get the Jeffreys prior

    ( , ) = det( I ( , )) 1

    | |p +1

    2

    17/22

    Multivariate Normal Distribution

    http://find/
  • 8/12/2019 Bayesian Gaussian

    18/22

    Consequences of the use of the Jeffreys prior is positive denite and symmetric. Using the Spectral Dec.Thm one can write as = H DH , where H is a matrix whosecolumns are the eigenvectors and D is the diagonal matrix witheigenvalues in a non increasing order

    H H = I p D = diag (1, . . . , p).

    Then, assuming that all the eigenvalues are different,

    ( )d = (H , D )I [ 1 > 2 > > p ]dH dD

    With a change of variable,

    (H , D ) = (H DH )i

  • 8/12/2019 Bayesian Gaussian

    19/22

    Then

    J (, ) = J (, H , D ) 1D |

    p +12 i

  • 8/12/2019 Bayesian Gaussian

    20/22

    Gibbs sampling for N ( , )

    Also in the multivariate case, it can be useful and convenient

    to adopt a computational approach rather then perform closedform calculations.this solution is particularly important when you are interestedin functions of and .

    20/22

    Multivariate Normal Distribution

    http://find/
  • 8/12/2019 Bayesian Gaussian

    21/22

    Full conditionals

    We need to write down the two full conditionals that is | , x | , x .

    The rst one is already known.

    | , x N p( , (c + n) 1 ), (2)

    The second one can be easily seen to be

    | , x IW p m + n, 1 +

    n

    i=1(x i )(x i ) (3)

    21/22

    Multivariate Normal Distribution

    http://find/
  • 8/12/2019 Bayesian Gaussian

    22/22

    R code

    22/22

    http://find/