-
Smoothing and Forecasting
Mortality Rates with P-splines
Iain CurrieHeriot Watt University
London, June 2006
-
Data and problem
• Data: CMI assured lives
– Age: 20 to 90
– Year: 1947 to 2002
• Problem: forecast table to 2046
-
Data and problem
• Data: CMI assured lives
– Age: 20 to 90
– Year: 1947 to 2002
• Problem: forecast table to 2046
-
Plan of talk
• P-splines in 1-dimension
• P-splines in 2-dimensions
– Lee-Carter model
– Age-Period-Cohort model
– 2-d P-splines
-
Plan of talk
• P-splines in 1-dimension
• P-splines in 2-dimensions
– Lee-Carter model
– Age-Period-Cohort model
– 2-d P-splines
-
20 30 40 50 60 70 80 90
−10
−8
−6
−4
−2
Gompertz model
Age
Log(
mor
talit
y)
Year 2002
Gompertz: log m = −11.73 + 0.109 Age
-
1950 1960 1970 1980 1990 2000
−4.
0−
3.8
−3.
6−
3.4
−3.
2
Simple Gompertz
Year
Log(
mor
talit
y)
Age 70
log m = 26.9 − 0.0154 Year
-
1950 1960 1970 1980 1990 2000
−4.
0−
3.8
−3.
6−
3.4
−3.
2
Quadratic Gompertz
Year
Log(
mor
talit
y)
Age 70
log m = 26.9 − 0.0154 Year
log m = − 1100 + 1.127 Year − 0.000289 Year^2
-
1960 1980 2000 2020 2040
−6.
0−
5.5
−5.
0−
4.5
−4.
0−
3.5
−3.
0
Quadratic forecast
Year
Log(
mor
talit
y)
Age 70
-
1960 1980 2000 2020 2040
−6.
0−
5.5
−5.
0−
4.5
−4.
0−
3.5
−3.
0
Quadratic fit: linear & quadratic forecasts
Year
Log(
mor
talit
y)
Age 70
Quadratic forecast
Linear forecast
-
Regression basisIn ordinary regression we use basis like
Linear basis: {1, x}
Quadratic basis: {1, x, x2}
Polynomial basis: {1, x, x2, . . . , xp}.
Means and fitted values are linear combinations of the basis
functions
logµ = a+ bx, log µ̂ = â+ b̂x
logµ = a+ bx+ cx2, log µ̂ = â+ b̂x+ ĉx2
logµ = a+ b1x+ . . .+ bpxp, log µ̂ = â+ b̂1x+ . . .+ b̂px
p
-
Regression basisIn ordinary regression we use basis like
Linear basis: {1, x}
Quadratic basis: {1, x, x2}
Polynomial basis: {1, x, x2, . . . , xp}.
Means and fitted values are linear combinations of the basis
functions
logµ = a+ bx, log µ̂ = â+ b̂x
logµ = a+ bx+ cx2, log µ̂ = â+ b̂x+ ĉx2
logµ = a+ b1x+ . . .+ bpxp, log µ̂ = â+ b̂1x+ . . .+ b̂px
p
-
Generalised linear modelsModel fitting uses the Generalised
Linear Model framework.
dx ∼ P(Ecxµx)
and logEcxµx = logEcx + a+ bx
or logEcxµx = logEcx + a+ bx+ cx
2.
-
B-spline basisA B-spline regression basis uses local basis
functions.
B-spline basis: {B1(x), B2(x), . . . , Bp(x)}
where B1(x), B2(x), . . . , Bp(x) are B-splines.
Means and fitted values work the same way
logµ =
p∑
1
θjBj(x), log µ̂ =
p∑
1
θ̂jBj(x)
-
B-spline basisA B-spline regression basis uses local basis
functions.
B-spline basis: {B1(x), B2(x), . . . , Bp(x)}
where B1(x), B2(x), . . . , Bp(x) are B-splines.
Means and fitted values work the same way
logµ =
p∑
1
θjBj(x), log µ̂ =
p∑
1
θ̂jBj(x)
-
20 40 60 80 100
0.0
0.1
0.2
0.3
0.4
0.5
0.6
A single cubic B−spline
Age
B−
splin
e
-
20 40 60 80 100
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Cubic B−spline basis
Age
B−
splin
e
-
1950 1960 1970 1980 1990 2000
−4.
0−
3.8
−3.
6−
3.4
−3.
2
B−spline regression
Year
log(
Mor
talit
y)
Age 70
Number of B−splines: 23
-
1950 1960 1970 1980 1990 2000
−4.
0−
3.8
−3.
6−
3.4
−3.
2
B−spline regression
Year
log(
Mor
talit
y)
Age 70
Number of B−splines: 23
-
PenaltiesEilers & Marx (1996) imposed penalties on
differences between adjacent
coefficients.
P2(θ) = (θ1 − 2θ2 + θ3)2 + . . .+ (θp−2 − 2θp−1 + θp)
2
= θ′D′2D2θ
where D2 is a difference matrix.
P2(θ) is a roughness penalty.
Estimation is via penalised likelihood
PL(θ) = L(θ)− 12λθ′D′2D2θ
where λ is the smoothing parameter which balances fit and
smoothness.
• Interpolation (λ = 0)
• Linear regression (λ = ∞)
-
PenaltiesEilers & Marx (1996) imposed penalties on
differences between adjacent
coefficients.
P2(θ) = (θ1 − 2θ2 + θ3)2 + . . .+ (θp−2 − 2θp−1 + θp)
2
= θ′D′2D2θ
where D2 is a difference matrix.
P2(θ) is a roughness penalty.
Estimation is via penalised likelihood
PL(θ) = L(θ)− 12λθ′D′2D2θ
where λ is the smoothing parameter which balances fit and
smoothness.
• Interpolation (λ = 0)
• Linear regression (λ = ∞)
-
PenaltiesEilers & Marx (1996) imposed penalties on
differences between adjacent
coefficients.
P2(θ) = (θ1 − 2θ2 + θ3)2 + . . .+ (θp−2 − 2θp−1 + θp)
2
= θ′D′2D2θ
where D2 is a difference matrix.
P2(θ) is a roughness penalty.
Estimation is via penalised likelihood
PL(θ) = L(θ)− 12λθ′D′2D2θ
where λ is the smoothing parameter which balances fit and
smoothness.
• Interpolation (λ = 0)
• Linear regression (λ = ∞)
-
1950 1960 1970 1980 1990 2000
−4.
0−
3.8
−3.
6−
3.4
−3.
2
P−spline regression
Year
log(
Mor
talit
y)
Age 70
Number of B−splines: 23
Quadratic penalty
-
1960 1980 2000 2020 2040
−5.
0−
4.5
−4.
0−
3.5
−3.
0
Forecasting to 2046
Year
log(
Mor
talit
y)
Age 70
Forecast to 2046
Quadratic penalty: linear forecast
-
1960 1980 2000 2020 2040
−5.
0−
4.5
−4.
0−
3.5
−3.
0
Forecasting to 2046
Year
log(
Mor
talit
y)
Age 70
Delta knots = 2.75
Delta knots = 11
-
Lee-Carter modelLee & Carter (1992)
logµij = αi + βiκj , 20 ≤ i ≤ 90, 1947 ≤ j ≤ 2002
-
20 30 40 50 60 70 80 90
−7
−6
−5
−4
−3
−2
Age
Alp
ha
20 30 40 50 60 70 80 90
1.5
2.0
2.5
Age
Bet
a
1950 1960 1970 1980 1990 2000
−0.
3−
0.2
−0.
10.
00.
1
Year
Kap
pa
1950 1960 1970 1980 1990 2000
−4.
0−
3.8
−3.
6−
3.4
−3.
2
Year
log(
mor
talit
y)
Age: 70
-
1950 1960 1970 1980 1990 2000
−6.
2−
5.8
−5.
4
Year
log(
mor
talit
y)
Age: 50
1950 1960 1970 1980 1990 2000
−5.
2−
4.8
−4.
4
Year
log(
mor
talit
y)
Age: 60
1950 1960 1970 1980 1990 2000
−4.
0−
3.8
−3.
6−
3.4
−3.
2
Year
log(
mor
talit
y)
Age: 70
1950 1960 1970 1980 1990 2000
−2.
8−
2.6
−2.
4−
2.2
Year
log(
mor
talit
y)
Age: 80
-
1960 1980 2000 2020 2040
−8
−6
−4
−2
Lee−Carter forecasts to 2046
Year
log(
mor
talit
y)
40
50
60
70
80
90
-
Discrete Lee-Carter modellogµij = αi + βiκj , 20 ≤ i ≤ 90, 1947
≤ j ≤ 2002
or in table form
logM = α1′ + βκ′
Smooth Lee-Carterα→ Baa, β → Bab, κ→ Byk
-
Discrete Lee-Carter modellogµij = αi + βiκj , 20 ≤ i ≤ 90, 1947
≤ j ≤ 2002
or in table form
logM = α1′ + βκ′
Smooth Lee-Carterα→ Baa, β → Bab, κ→ Byk
-
20 30 40 50 60 70 80 90
−8
−7
−6
−5
−4
−3
−2
Age
Alp
ha
20 30 40 50 60 70 80 90
56
78
9
Age
Bet
a
1960 1980 2000 2020 2040
−0.
20−
0.10
0.00
0.10
Year
Kap
pa
1960 1980 2000 2020 2040
−5.
0−
4.5
−4.
0−
3.5
Year
log(
mor
talit
y)
Age: 70
-
1960 1980 2000 2020 2040
−8
−6
−4
−2
Smooth Lee−Carter: ages 40 to 90
Year
Log(
mor
talit
y
40
50
60
70
80
90
-
Age-Period-Cohort modellogµij = αi + βj + γi−j , 20 ≤ i ≤ 90,
1947 ≤ j ≤ 2002
Parameter redundancy• 2na + 2ny − 1 = 253 parameters
• 2na + 2ny − 4 = 250 free parameters
-
Age-Period-Cohort modellogµij = αi + βj + γi−j , 20 ≤ i ≤ 90,
1947 ≤ j ≤ 2002
Parameter redundancy• 2na + 2ny − 1 = 253 parameters
• 2na + 2ny − 4 = 250 free parameters
-
1950 1960 1970 1980 1990 2000
−6.
2−
6.0
−5.
8−
5.6
−5.
4−
5.2
Year
log(
mor
talit
y)
Age: 50
1950 1960 1970 1980 1990 2000
−5.
0−
4.8
−4.
6−
4.4
−4.
2
Year
log(
mor
talit
y)
Age: 60
1950 1960 1970 1980 1990 2000
−4.
0−
3.8
−3.
6−
3.4
−3.
2
Year
log(
mor
talit
y)
Age: 70
1950 1960 1970 1980 1990 2000
−3.
0−
2.8
−2.
6−
2.4
−2.
2
Year
log(
mor
talit
y)
Age: 80
-
Discrete Age-Period-Cohort modellogµij = αi + βj + γi−j , 20 ≤ i
≤ 90, 1947 ≤ j ≤ 2002
Smooth Age-Period-Cohort modelα→ Baa, β → Byb, γ → Bcc
-
Discrete Age-Period-Cohort modellogµij = αi + βj + γi−j , 20 ≤ i
≤ 90, 1947 ≤ j ≤ 2002
Smooth Age-Period-Cohort modelα→ Baa, β → Byb, γ → Bcc
-
1950 1960 1970 1980 1990 2000
−6.
2−
6.0
−5.
8−
5.6
−5.
4−
5.2
Year
log(
mor
talit
y)
Age: 50
1950 1960 1970 1980 1990 2000
−5.
0−
4.8
−4.
6−
4.4
−4.
2
Year
log(
mor
talit
y)
Age: 60
1950 1960 1970 1980 1990 2000
−4.
0−
3.8
−3.
6−
3.4
−3.
2
Year
log(
mor
talit
y)
Age: 70
1950 1960 1970 1980 1990 2000
−3.
0−
2.8
−2.
6−
2.4
−2.
2
Year
log(
mor
talit
y)
Age: 80
-
AgeYear
Log(mortality)
Planar component
20 30 40 50 60 70 80 90
−1.
0−
0.5
0.0
0.5
Age component
Age
Log(
mor
talit
y)
1950 1960 1970 1980 1990 2000
−0.
06−
0.02
0.02
0.06
Year component
Year
Log(
mor
talit
y)
1860 1880 1900 1920 1940 1960 1980
−0.
15−
0.05
0.05
Cohort component
Cohort
Log(
mor
talit
y)
-
20 30 40 50 60 70 80 90
−8
−7
−6
−5
−4
−3
−2
Components of Age
Age
Log(
mor
talit
y
194719742002
Improvement = 0.015/annum
-
1950 1960 1970 1980 1990 2000
−7
−6
−5
−4
−3
−2
Components of Year
Year
Log(
mor
talit
y
Age 20
Age 90
Improvement = 0.7/10years
-
1860 1880 1900 1920 1940 1960 1980
−7
−6
−5
−4
−3
−2
Components of Cohort
Cohort
Log(
mor
talit
y
185718711885
Improvement = 0.29/10years
-
2-d modelling of mortality tables• Let Ba, 71× 13, be a 1-d
B-spline basis for modelling a single age.
• Let By, 56× 10, be a 1-d B-spline basis for modelling a single
year.
Can we combine the marginal bases Ba and By to make a 2-d
basis?
-
2-d modelling of mortality tables• Let Ba, 71× 13, be a 1-d
B-spline basis for modelling a single age.
• Let By, 56× 10, be a 1-d B-spline basis for modelling a single
year.
Can we combine the marginal bases Ba and By to make a 2-d
basis?
-
1947
1960
1973
1986
1999
Year
11
33
56
78
100
Age
00.
10.
20.
30.
40.
5
-
2-d modelling of mortality tablesRegression matrix
The Kronecker product organises the multiplication of the
marginal bases to
give the regression matrix
B = By ⊗Ba, 3976× 130.
Penalties in 2-d
• Each regression coefficient is associated with the summit of
one of the hills.
• Smoothness is ensured by penalizing the coefficients in rows
and columns.
-
2-d modelling of mortality tablesRegression matrix
The Kronecker product organises the multiplication of the
marginal bases to
give the regression matrix
B = By ⊗Ba, 3976× 130.
Penalties in 2-d
• Each regression coefficient is associated with the summit of
one of the hills.
• Smoothness is ensured by penalizing the coefficients in rows
and columns.
-
1960 1980 2000 2020 2040
−7.
5−
7.0
−6.
5
Year
Log(
mor
talit
y
Age: 40
1960 1980 2000 2020 2040
−7.
0−
6.5
−6.
0−
5.5
Year
Log(
mor
talit
y
Age: 50
1960 1980 2000 2020 2040
−6.
0−
5.5
−5.
0−
4.5
−4.
0
Year
Log(
mor
talit
y
Age: 60
1960 1980 2000 2020 2040
−5.
0−
4.5
−4.
0−
3.5
Year
Log(
mor
talit
y
Age: 70
-
1960 1980 2000 2020 2040
−7
−6
−5
−4
−3
−2
2−d mortality: ages 40 to 90
Year
Log(
mor
talit
y
40
50
60
70
80
90
-
Summary of results
DEV TR BIC Rank
2-d 6832 63 7357 1
APC smooth 7852 38 8167 2
LC smooth 8568 30 8815 3
APC 7002 248 9057 4
LC 8004 196 9629 5
-
1960 1980 2000 2020 2040
−7.
0−
6.5
−6.
0−
5.5
−5.
0−
4.5
−4.
0
Summary for age 60
Year
log(
mor
talit
y)
APC2−dLCSmoothLC
-
1960 1980 2000 2020 2040
−5.
5−
5.0
−4.
5−
4.0
−3.
5
Summary for age 70
Year
log(
mor
talit
y)
APC2−dLCSmoothLC
-
1960 1980 2000 2020 2040
−4.
0−
3.5
−3.
0−
2.5
Summary for age 80
Year
log(
mor
talit
y)
APC2−dLCSmoothLC
-
1960 1980 2000 2020 2040
−7
−6
−5
−4
−3
−2
Summary for ages 60, 70, 80
Year
log(
mor
talit
y)
60
70
80
APC2−dLCSmoothLC
-
ConclusionsForecasting a mortality table depends on
• Model choice
• Forecasting method
• Parameter uncertainty
• Stochastic uncertainty
-
ConclusionsForecasting a mortality table depends on
• Model choice
• Forecasting method
• Parameter uncertainty
• Stochastic uncertainty
-
ConclusionsForecasting a mortality table depends on
• Model choice
• Forecasting method
• Parameter uncertainty
• Stochastic uncertainty
-
ConclusionsForecasting a mortality table depends on
• Model choice
• Forecasting method
• Parameter uncertainty
• Stochastic uncertainty
-
ConclusionsForecasting a mortality table depends on
• Model choice
• Forecasting method
• Parameter uncertainty
• Stochastic uncertainty
-
References
Lee-Carter models
Lee & Carter (1992) J American Statistical Association, 87,
659-675.
Brouhns, Denuit & Vermunt (2002) Insurance: Mathematics
& Economics, 31,
373-393.
Age-Period-Cohort models
Clayton & Schlifflers (1987) Statistics in Medicine, 6,
449-467.
Clayton & Schlifflers (1987) Statistics in Medicine, 6,
469-481.
Penalized spline models
Eilers & Marx (1996) Statistical Science, 11, 758-783.
Currie, Durban & Eilers (2004) Statistical Modelling, 4,
279-298.
Currie, Durban & Eilers (2006) Journal of the Royal
Statistical Society, Series
B, 68, 259-280.
