Smoothing mortality rates using R

Smoothing mortality rates using R

Gary Brown & Julie Mills

Overview

• Introduction• Context• Methodology• Implementation• Results• Summary• What’s next?

Introduction

• Future mortality rates are published every two years – Until 2004, by the Government Actuary’s

Department (GAD)– Since 2006, by ONS (with GAD consultants)

• The methodology was designed by GAD in the 1990s and runs in Excel

• In 2010, ONS reviewed the current process– Implementation and testing still ongoing

Context

• Mortality rates, estimated 75 years into the future, are a key factor in National Population Projections (others: births and net migration)– Natural change (births – deaths) accounts for 1/3

of total population change

• Population projections used as inputs/control totals for other government projections, such as numbers of school children or pensioners

• Robustness of mortality rates is crucial

Methodology - current

Mortality rate = deaths/pop

Methodology - current

Mortality rate = deaths/popconstrained survivor ratio

Methodology - current

Mortality rate = deaths/pop

Smooth within years (to 103/104)

constrained survivor ratio

Methodology - current

Mortality rate = deaths/pop

Smooth within years (to 103/104)

constrained survivor ratio

extrapolate to age 120

Methodology - current

Mortality rate = deaths/pop

Smooth within years (to 103/104)

Estimate year T+1 for each age

constrained survivor ratio

extrapolate to age 120

Methodology - current

Mortality rate = deaths/pop

Smooth within years (to 103/104)

Estimate year T+1 for each age

constrained survivor ratio

extrapolate to age 120

exponential smoothingx2

Methodology - current

Mortality rate = deaths/pop

Smooth within years (to 103/104)

Estimate year T+1 for each age

Smooth improvement rate in T+1

constrained survivor ratio

extrapolate to age 120

exponential smoothingx2

Methodology - current

Mortality rate = deaths/pop

Smooth within years (to 103/104)

Estimate year T+1 for each age

Smooth improvement rate in T+1

constrained survivor ratio

extrapolate to age 120

exponential smoothingx2

1x1 3x1 5x1 3x1 1x1 MAs

Methodology - current

Mortality rate = deaths/pop

Smooth within years (to 103/104)

Estimate year T+1 for each age

Smooth improvement rate in T+1

Improvement rates up to T+26

constrained survivor ratio

extrapolate to age 120

exponential smoothingx2

1x1 3x1 5x1 3x1 1x1 MAs

Methodology - current

Mortality rate = deaths/pop

Smooth within years (to 103/104)

Estimate year T+1 for each age

Smooth improvement rate in T+1

Improvement rates up to T+26

constrained survivor ratio

extrapolate to age 120

exponential smoothingx2

1x1 3x1 5x1 3x1 1x1 MAs

T+26 expert opinions

Methodology - current

Mortality rate = deaths/pop

Smooth within years (to 103/104)

Estimate year T+1 for each age

Smooth improvement rate in T+1

Improvement rates up to T+26

Mortality rates for T+1 to T+26

constrained survivor ratio

extrapolate to age 120

exponential smoothingx2

1x1 3x1 5x1 3x1 1x1 MAs

T+26 expert opinions

Methodology - current

Mortality rate = deaths/pop

Smooth within years (to 103/104)

Estimate year T+1 for each age

Smooth improvement rate in T+1

Improvement rates up to T+26

Mortality rates for T+1 to T+26

constrained survivor ratio

extrapolate to age 120

exponential smoothingx2

1x1 3x1 5x1 3x1 1x1 MAs

T+26 expert opinions

… further adjustments

Methodology - current

Mortality rate = deaths/pop

Smooth within years (to 103/104)

Estimate year T+1 for each age

Smooth improvement rate in T+1

Improvement rates up to T+26

Mortality rates for T+1 to T+26

constrained survivor ratio

extrapolate to age 120

exponential smoothingx2

1x1 3x1 5x1 3x1 1x1 MAs

T+26 expert opinions

… further adjustments

Methodology - new

• Replace two-stage smoothing process

• Smooth mortality rates surface simultaneously over ages and years

• Estimate improvement rate using existing smoothed years – ie do not estimate T+1

• Requires longer path to T+26 opinions!

Methodology – 2 dimensional p-spline

• Thoroughly tested, and recommended, by Continuous Mortality Investigation

Methodology – 2 dimensional p-spline

• Thoroughly tested, and recommended, by Continuous Mortality Investigation

T Ta r r r y c c c

a y ages years

t t+5

PL( )=L( )-0.5 P( )

where P( )= I D D + I D D

where D , D act on rows/columns of B B

where B is matrix of B-splines, each fitting (x , x )

Methodology – 2 dimensional p-spline

• Thoroughly tested, and recommended, by Continuous Mortality Investigation

T Ta r r r y c c c

a y ages years

t t+5

PL( )=L( )-0.5 P( )

where P( )= I D D + I D D

where D , D act on rows/columns of B B

where B is matrix of B-splines, each fitting (x , x )

Methodology – 2 dimensional p-spline

• Thoroughly tested, and recommended, by Continuous Mortality Investigation

• Best advice - read “Smoothing and forecasting mortality rates”, Currie et al, 2004!

T Ta r r r y c c c

a y ages years

t t+5

PL( )=L( )-0.5 P( )

where P( )= I D D + I D D

where D , D act on rows/columns of B B

where B is matrix of B-splines, each fitting (x , x )

Implementation

• Difficult to understand (and explain) … but easy to implement!

Implementation

• Difficult to understand (and explain) … but easy to implement!

• MortalitySmooth (Carlo G Camarda) in R

Implementation

• Difficult to understand (and explain) … but easy to implement!

• MortalitySmooth (Carlo G Camarda) in R

Mort2Dsmooth(x=ages,y=years,Z=deaths,offset=log(pop))

Implementation

• Difficult to understand (and explain) … but easy to implement!

• MortalitySmooth (Carlo G Camarda) in R

• Smoothed values = 21st entry in list of R output

Mort2Dsmooth(x=ages,y=years,Z=deaths,offset=log(pop))

Results - testing

Males Females

Sensitivity – adding annual data

Ages 0 - 1031961 – 2003

1961 – 2004

1961 – 2005

1961 – 2006

1961 – 2007

1961 – 2008

1961 – 2009

Ages 0 - 1041961 – 2003

1961 – 2004

1961 – 2005

1961 – 2006

1961 – 2007

1961 – 2008

1961 – 2009

Sensitivity

- adding years of age

1961 – 20090 – 90

0 – 100

0 – 103

1961 – 2005 1961 - 2009 0 – 104 0 – 90

0 – 105 0 – 100

0 – 106 0 – 104

0 – 105

0 – 106

Results - testing

0

1

2

3

4

5

6

102

%

Age

0 20 40 60 8010 30 50 70 90

61-04

61-05

61-06

61-09

Mortality improvement rates by age, 2003/04

61-07

61-08

Results – mortality rates in the base year

Issues• New method does not project rates forward to

base year• Edge effects

Solution• Step back 2 years into the data set

2010 mortality rates = 2007 mortality rates x (1 – 2006-07 improvement rates/100) ^

3

Results – mortality rates in the base year

Year

Age

100

52 yrs

49

48 48 yrs

0

1961………………………………………………….. 2007 2008 2009 2010

Past improvements in smoothed mortality rates, males – old method

Past improvements in smoothed mortality rates, males – new method

Past improvements in smoothed mortality rates, Scotland males – new method

Past improvements in smoothed mortality rates, females – old method

Past improvements in smoothed mortality rates, females – new method

Comparison of projected smooth % changes in death rates by age, UK 2009-10

Males

-1

0

1

2

3

4

5

6

0 10 20 30 40 50 60 70 80 90 100 110

Age

Per

cent

age

redu

ctio

n

2008-basedprojections

Proposed 2010-based projections

Comparison of projected smooth % changes in death rates by age, UK 2009-10

Females

0

1

2

3

4

5

6

0 10 20 30 40 50 60 70 80 90 100 110

Age

2008-basedprojections

Proposed 2010-based projections

Comparison of actual and projected expectation of life at birth

65

70

75

80

85

90

1980 1985 1990 1995 2000 2005 2010 2015 2020 2025 2030 2035 2040 2045 2050 2055 2060 2065 2070 2075 2080 2085

Old 08

New 08

Old 08

New 08

Females

Males

Summary

• New smoothing method used to produce the 2010-based ‘proposed’ mortality assumptions

• Introduced in the 2010-based consultation with devolved administrations and government departments

What’s next?

• More testing/evaluation:Over-smoothingAdding 2010 dataDerivation of base year rates

• Using R to project mortality rates