Modeling biological-composition time series in integrated stock assessments: data weighting considerations and impact on estimates of stock status P. R

Modeling biological-composition time series in integrated stock assessments:

data weighting considerations and impact on estimates of stock status

P. R. CroneSouthwest Fisheries Science Center (NOAA)

Center for the Advancement of Population Assessment Methodology (CAPAM)8901 La Jolla Shores Dr., La Jolla, CA 92037, USA

0%

10%

20%

30%

40%

50%

Percent

1 2 3 4 5 6 7 8 9+

1966

1967

1968

. . .

2004

2005

2006

Age

Year

0%

10%

20%

30%

40%

50%

Percent

1 2 3 4 5 6 7 8 9+

1966

1967

1968

. . .

2004

2005

2006

Age

Year

Fishery 1

Fishery 2

•Study description•Results•Conclusions•Further work

Presentation outline

•Motivation and expectationso Better understanding of impact that data weighting considerations

in typical assessments have on baseline management statistics …contribute to good practices for stock assessment development

o Meta-analysis is based on a limited pool of assessments …is able to provide quantitative results for particular statistical comparisons, is not a substitute for simulation-based tests

Study description

•Assessment archiveoPool of recently conducted fish stock (species) assessments used for managementoAssessments for small pelagic (3), large pelagic (7), and groundfish (19) speciesoAssessments based on the Stock Synthesis modeloMajority of assessments conducted in 2015, some 2011-14

•Biological-composition time seriesoLength (‘marginal’, e.g., no./pct. by length bin and time step)oAge (marginal) oConditional age-at-length (‘random at length’, age-length key format)oSize (marginal, e.g., weight, biological compositions based on different bin structure) oWeight (unfitted empirical weight-at-age data) oVarious ways of using/combining biological-composition time series in assessments

GeneralStudy description

General (continued)Study description

•Data weighting of biological compositions ‘outside’ the modelo Initial (input) sample sizes for biological compositions are assessment/analyst-specific

o Sometimes based on actual number of fish (e.g., sport fishery compositions, CAAL)

o More often based on number of boat trips, hauls, sets, wells, sample adjustment formula, etc.

o Can be based generally on variance estimates determined from sample/survey programs

o Can be based generally on variance estimates from simulation analysis (e.g., bootstrap methods)

o Often caps (thresholds) are used for input sample sizes (e.g., 100-200)

o Input sample size determination was not addressed in this evaluation

General (continued)Study description

•Data weighting of biological compositions ‘inside’ the modelo Variability of biological-composition time series is based initially on input sample size … subsequently,

adjusted internally based on comparing observed and expected values from fits to the time series

o Various data weighting approaches for composition time series in integrated assessment models … McCallister and Ianelli (1997) and Francis (2011) methods often considered in practice

o ‘Effective’ sample size in Stock Synthesis model (McCallister and Ianelli methods) reflects number of random samples (drawn from multinomial distribution) needed to produce fit as precise as model’s predicted fit

o Actual weighting values (scalars) for composition data reflect various mean estimates calculated from ratios of effective to input sample sizes (multiplicative based)

o Francis method basis is variation of mean length/age of the composition time series, accounts for correlation among length or age groups, results in greater variation surrounding composition time series

o In practice, ad hoc caps (thresholds) are implemented for estimated scalars >1

o Internally implemented data weighting methods for composition time series were addressed in this evaluation

• Baseline (Final)o Assessment model for advising management

• Unweighted (UW)o Final model that includes no (internally) weighted composition time serieso All scalars (‘weighting values, variance adjustments, lambdas’) = 1

• McCallister-Ianelli (AM)o Scalar estimate reflects arithmetic mean from model fits to composition time series (based on

ratios of effective sample size to input sample sizes)• McCallister-Ianelli (HM)

o Scalar estimate reflects harmonic mean from model fits to composition time series (based on ratios of effective sample size to input sample sizes)

• Francis (F0)o Assessments that included only length- and/or age-composition time series and no CAAL time

series (based on FA)• Francis-Method A (FA)

o Assessments that included CAAL time series along with length and/or age-composition time series (mean estimates indexed by year)

• Francis-Method B (FB)o Assessments that included CAAL time series along with length and/or age-composition time

series (mean estimates indexed by year/length bin)

Study descriptionAssessment models Data weighting methods

•Model development/estimationo For each species, final assessment model re-configured according to recommended scalars from

respective data weighting method (cap=100 and single iteration)

o For a species, from 3-5 alternative models were developed for overall study, depending on the biological compositions, SS version, convergence issues

o Data weighting addressed only biological compositions included in the model, i.e., no weighting applicable to other input data (e.g., index of abundance time series) or parameter assumptions (e.g., σR of stock-recruit relationship)

o Data weighting methods described in McCallister and Ianelli (1997), Francis (2011), Methot and Wetzel (2013), Punt (in press)

•Outputo Management quantities of interest: MSY, FMSY, Bcurrent, Depletion (SSBcurrent / SSB0)o Comparisons based on means/CVs and medians/REs

Study descriptionGeneral (continued)

Data weighting methods – Example (SS effective sample size)

Data weighting method (scalar calculation)McCallister and Ianelli (AM)Mean (Input N) / Mean (Effective N) = 22.27 / 85.27 = 0.26Mean (Input N / Effective N) = 0.33

McCallister and Ianelli (HM)Mean (Input N) / Mean (Effective N) = 9.95 / 85.27 = 0.12Mean (Input N / Effective N) = 0.12

Fishery_Survey Year Input N (sample size) Effective N (sample size) Effective N / Input N1 1983 106.72 10.51 0.101 1984 91.64 16.97 0.191 1985 104.24 5.64 0.051 1986 120.00 15.79 0.131 1987 165.16 15.56 0.091 1988 179.08 57.87 0.321 1989 143.32 3.31 0.021 1990 84.56 17.06 0.201 1991 66.20 10.33 0.161 1992 79.76 49.40 0.621 1993 107.52 38.04 0.351 1994 124.56 26.31 0.211 1995 108.24 9.51 0.091 1996 87.56 110.99 1.271 1997 108.56 30.41 0.281 1998 90.20 50.16 0.561 1999 66.64 49.55 0.741 2000 76.40 10.37 0.141 2001 84.44 3.82 0.051 2002 85.80 2.84 0.031 2003 62.80 21.12 0.341 2004 101.16 3.59 0.041 2005 91.96 14.13 0.151 2006 95.72 18.89 0.201 2007 64.36 29.81 0.461 2008 28.92 21.16 0.731 2009 16.88 3.43 0.201 2010 19.88 20.18 1.011 2011 30.84 8.61 0.281 2012 47.80 6.68 0.141 2013 71.72 8.41 0.121 2014 16.00 22.21 1.39

MeansArithmetic 85.27 22.27 0.33Harmonic 9.95 0.12

Mea

n le

ngth

(cm

)

Year5

Data weighting methods – Example (FA/FB/F0 diagnostic plot)

Species

BaselineFinal

Assessment model(Data weighting method)

Assessment model(Final)

P. sardine

Study descriptionAnalysis flow chart

McCallister/Ianelli (Harmonic mean)HM

McCallister/Ianelli (Arithmetic mean) AM

Unweighted (All scalars=1)UW

Francis (No CAAL)F0

Francis (CAAL, Method A)FA

Francis (CAAL, Method B)FB

Output(Management quantities)

MSY, FMSY, Bcurrent, DEP






.

.

.

.

.

.

.N = 29 species

Results

Data Weighting MethodsScalar Ranges by Biological Data Type

Low High Low High Low HighMcCallister-Ianelli (AM) 0.03 53.91 0.21 8.35 0.65 >1000McCallister-Ianelli (HM) 0.01 11.38 0.12 5.32 0.03 >1000Francis (F0) 0.004 10.13 0.04 >1000 FA 0.02 >1000 FB 0.13 >1000

Data weighting method LengthBiological data

Age CAAL

Assessment (species) examplesLength Age CAAL MSY (mt) Fmsy Bcurrent (mt) Depletion

P. sardine (S3) Final (F) X X 63,212 0.29 151,968 0.23P. sardine (S3) Weight_None (UW) 77,812 0.28 156,987 0.19P. sardine (S3) Weight_McCallister-Ianelli (AM) 81,534 0.27 134,320 0.15P. sardine (S3) Weight_McCallister-Ianelli (HM) 89,337 0.27 168,972 0.18P. sardine (S3) Weight_Francis (F0) na na na naP. sardine (S3) Weight_Francis (FA) 116,720 0.25 240,463 0.21P. sardine (S3) Weight_Francis (FB) 112,021 0.24 213,398 0.20

N.P. albacore (L1) Final (F) X 105,278 0.23 669,405 0.36N.P. albacore (L1) Weight_None (UW) 134,188 0.23 734,324 0.32N.P. albacore (L1) Weight_McCallister-Ianelli (AM) 110,451 0.23 702,207 0.35N.P. albacore (L1) Weight_McCallister-Ianelli (HM) 104,707 0.23 650,254 0.34N.P. albacore (L1) Weight_Francis (F0) 105,401 0.23 666,337 0.34N.P. albacore (L1) Weight_Francis (FA) na na na naN.P. albacore (L1) Weight_Francis (FB) na na na na

Composition time series Management quantitySpecies Data weighting method

Mean and CV

Mean and CV

Data Weighting Methods‘Within Assessment’ Variability

MSY

G1 G2 G3 G4 G5 G6 G7 G8 G9 G10G11G12G13G14G15G16G17G18G19L1 L2 L3 L4 L5 L6 L7 S1 S2 S3

0

25

50

75

100

Assmt. CV (MSY)

Assessment (species)

CV (%)+


FMSY


0

25

50

75

Assmt. CV (Fmsy)


CV (%)


Bcurrent


0

25

50

75

100

Assmt. CV (Bcurrent)


CV (%)+


Depletion


0

20

40

60

80

Assmt. CV (Depletion)


CV (%)

Data Weighting Methods‘Between Management Quantity’ Variability

0

5

10

15

20

25

30

35

40

45

50

55

Mgt. qty. CV

Management quantity

CV (%)

MSY BcurrentFMSY Depletion

Data weighting method

MSY

Species (no. of assessments) 29 27 29 11 15 15Models (no. of replicates) 126 119 126 43 74 74Sample size limit implemented (no. of species) 0 2 0 0 1 1Convergence issues (no. of species) 0 2 0 0 0 0Unplotted models (pct. extreme positive outliers) 4% 2% 2% 2% 4% 1%

‘Within Assessment’ Variability (Relative to Data Weighting Method)

Rel

ativ

e er

ror

FMSY



Rel

ativ

e er

ror


Bcurrent



Rel

ativ

e er

ror


Depletion



Rel

ativ

e er

ror


Data Weighting MethodsRelative to HM (‘correctly specified’ model)

Rel

ativ

e er

ror


MSY

Species (no. of assessments) 29 27 11 15 15Models (no. of replicates) 29 27 11 15 15Sample size limit implemented (no. of species) 0 2 0 1 1Convergence issues (no. of species) 0 2 0 0 0Unplotted models (pct. extreme positive outliers) 0% 0% 0% 0% 13%


Rel

ativ

e er

ror

Data weighting methodSpecies (no. of assessments) 29 27 11 15 15Models (no. of replicates) 29 27 11 15 15Sample size limit implemented (no. of species) 0 2 0 1 1Convergence issues (no. of species) 0 2 0 0 0Unplotted models (pct. extreme positive outliers) 3% 0% 0% 0% 0%

FMSY


Rel

ativ

e er

ror

Data weighting methodSpecies (no. of assessments) 29 27 11 15 15Models (no. of replicates) 29 27 11 15 15Sample size limit implemented (no. of species) 0 2 0 1 1Convergence issues (no. of species) 0 2 0 0 0Unplotted models (pct. extreme positive outliers) 3% 0% 0% 0% 13%

Bcurrent


Rel

ativ

e er

ror

Species (no. of assessments) 29 27 11 15 15Models (no. of replicates) 29 27 11 15 15Sample size limit implemented (no. of species) 0 2 0 1 1Convergence issues (no. of species) 0 2 0 0 0Unplotted models (pct. extreme positive outliers) 0% 0% 0% 0% 7%

Depletion


Conclusions• Data weighting methods impact on management quantities

o Terminal biomass estimates most uncertain in most cases (mean CV=35%), depletion and MSY less so (20%), and FMSY most precise (<10%)

o Positively-skewed, median-unbiased relative error distributions

o The harmonic mean-based McCallister-Ianelli method (HM) resulted in precise and unbiased estimates in most cases, but …

o Unweighted method (UW) also relatively precise and robust in many comparisons

o Frances methods (F0, FA, FB) produced generally unbiased estimates, but typically less precise than HM; more similar for MSY-related quantities

o FA less bias (equally precise) than FB in many comparisons

o For correctly-specified assessment based on HM, better off not weighting (UW) than implementing an alternative data weighting method

Study benefits and further work• Replicates (assessments) in meta-analysis are realistic

o Replicates associated with typical simulations are unrealistic, i.e., much too similar to one another … increase number/variety of assessments

o However, study (experimental) population based on real assessments provides limited cause-and-effect information, given the many data/parameter inconsistencies across replicates

• Meta-analysis provides baseline information for more focused simulation studieso Contrast between quality of derived management metricso Fold into MSEs addressing small pelagic species’ fisheries on the USA Pacific

coast for basing (much needed) new and improved harvest control rules• Information useful for analysts charged with developing

ongoing assessments for management purposeso Data weighting approaches in actual assessments are evolving presently,

research needed to inform good practices

ReferencesCrone, P.R., D.B. Sampson. 1998. Evaluation of assumed error structure in stock assessment models that use

sample estimates of age composition. Pages 355-370 in Fishery Stock Assessment Models. Alaska Sea Grant College Program Report No. AK-SG-98-01, University of Alaska, Fairbanks, Alaska.

Fournier, D., C.P. Archibald. 1982. A general theory for analyzing catch at age data. Can. J. Fish. Aquat. Sci. 39:1195-1207.

Francis, R.I.C.C. 2011. Data weighting in statistical fisheries stock assessment models. Can. J. Fish. Aquat. Sci. 68:1124-1138.

McAllister, M.K., J.N. Ianelli. 1997. Bayesian stock assessment using catch-age data and the sampling-importance resampling algorithm. Can. J. Fish. Aquat. Sci. 54(2): 284–300.

Methot, R.D., C.R. Wetzel. 2013. Stock Synthesis: a biological and statistical frame-work for fish stock assessment and fishery management. Fish. Res. 142:86–99.

Pennington, M., L.-M. Burmeister, V. Hjellvik. 2002. Assessing the precision of frequency distributions estimated from trawl survey samples. Fish Bull. 100:74–80.

Punt, A.E. in press. Some insights into data weighting in integrated stock assessments. Fish. Res.

Stewart, I.J., O.S. Hamel. 2014. Boostrapping of sample sizes for length- or age-composition data used in stock assessments. Can. J. Fish. Aquat. Sci. 671:581-588.

Documents

Modeling biological-composition time series in integrated stock assessments: data weighting considerations and impact on estimates of stock status P. R