Comparable Biological Assessmentsfrom Different Methods and Analyses

David B. Herbst1 and Erik L. Silldorff2

1 Sierra Nevada Aquatic Research Laboratory, University of California, Route 1-Box 198, Mammoth Lakes, CA 935462 Princeton Hydro, 1108 Old York Road, Suite 1, PO Box 720, Ringoes, NJ 08551

AbstractRegionalized bioassessment programs of state and federal regulatory agencies, and other government and private groups often use different methods to collect and analyze stream invertebrate samples. While this has created concern and confusion over the comparability of these many disparate sources of data, studies have only recently begun to evaluate differences in performance between methods and reconcile the results produced from different programs. To obtain directly comparable data sets, we conducted concurrent sampling at 40 sites in the eastern Sierra Nevada of California using three bioassessment methodologies that differed at each stage, from field sample collection to laboratory processing and data analysis (California Stream Bioassessment Protocol, Region 5 US Forest Service, and Lahontan Regional Water Board methods). We used a performance-based methods system to compare precision, uniformity, discrimination, accuracy, and correlations among multimetric and predictive model output assessment scores. Reference and test sites were first identified using local and upstream watershed disturbance criteria, and invertebrate community measures and models were then developed to discriminate between these site classes. Differences in performance between methods were small, and the assessment scores were both highly correlated and distinguished reference from test sites with similar accuracy. An examination of the association of impaired biological integrity with environmental stress gradients showed that the method using most replication and sample counts provided the best resolution of stressor effect thresholds and signal to noise ratio. Despite slight differences in performance and stress detection, these results demonstrate that even substantially different methods of bioassessment yield very similar, effective discrimination of impaired biological condition. Moreover, this conclusion did not depend on the data analysis approach since both multimetric IBIs and multivariate RIVPACS predictive models were in close agreement. These results suggest that data from multiple sources can often be standardized and used in integrated assessments of stream biological integrity.

Optimum IBIs for Each Method

0

20

40

60

80

100

UC - SNARL CDFG - CSBP USU - USFS.R5

6-metric 8-metric 7-metric

IBI S

co

re

Reference Sites

Test Sites

IBIs

0.0

1.0

2.0

3.0

4.0

5.0

UC - SNARL CDFG - CSBP USU - USFS.R5

6-metric 8-metric 7-metric

Rati

o

Signal

Signal:Noise

O/E Model for Each Method

0.0

0.2

0.4

0.6

0.8

1.0

1.2

UC - SNARL CDFG - CSBP USU - USFS.R5

(depth, lat, date) (elev, width, date) (area, days wet, date)

O/E

Valu

e

Reference Sites

Test Sites O/Es

0.0

1.0

2.0

3.0

4.0

5.0

UC - SNARL CDFG - CSBP USU - USFS.R5

(depth, lat, date) (elev, width, date) (area, days wet,date)

Rati

o

Signal

Signal:Noise

UC-SNARL CSBP USFS.R5

Net type, mesh D-frame, 250 μm D-frame, 500 μm D-frame, 500 μm

Replication 5 composites of 3 3 composites of 3 1 composite of 8

Area sampled 1.39 m2 1.67 m2 0.74 m2

Subsampling Drum splitter Grid tray Grid tray

Enumeration 250–500 count[total ~1500+]

300 fixed count[total 900]

500 fixed count[total 500]

Taxonomic resolution

Genus / species Genus / species (not midges & mites)

Genus / species

Contrast of basic differences between the riffle-based methods

Region & SettingEastern Sierra40 streams: 1st to 4th order6000-9000 ft elevImpact sources: Sediment•Livestock grazing•Channel alteration

0

2

4

6

8

10

12

UC-SNARL CSBP USFS-R5

Nu

mb

er

of

me

tric

s

20-25%

15-20%

10-15%

Metric Precision:Number of IBI metrics meeting potential DQO targets for the different methods -UC-SNARL with a few more than others

In press: JNABS Vol 25(2)

Responses to sediment stressor:Again similar but UC-SNARL with slightly improved detection of >60% threshold and less reference-test overlap

High correlations existed for site-by-site co-plots among methods. Similar reference/test distributions but lower variability in UC-SNARL method results in improved signal:noise ratio and potential power 0

25

50

75

100

0% 20% 40% 60% 80% 100%

%Fines-Sand-Gravel

SN

AR

L IB

I

0

25

50

75

100

0% 20% 40% 60% 80% 100%

%Fines-Sand-Gravel

CS

BP

IB

I

0

25

50

75

100

0% 20% 40% 60% 80% 100%

%Fines-Sand-Gravel

US

FS

-R5

IB

I

Between-method community similarity is low for CSBP relative to others because of low taxonomic resolution, but is close for SNARL x USFS and becomes nearly the same by re-sampling SNARL data to 500 fixed-count.

Despite slight performance advantage of UC-SNARL…..

Detection of impaired test sites equally effective and final site by site assessments are in agreement between methods in 83-93% of cases (based on criterion level at type I =17%) whether comparing within or between IBI or O/E scores

Data & output may be combined, OR….

Use integrated assessments and multiple lines of evidence in confirmation of results

0

25

50

75

100

S

SP TF

WO TC

CW EC

RH

CN

WW F

OS

WC L

SV T C

TP W D M CT

SL

WF

WP K R

TB B

OT

O4

RB

OA

OX

OP

OB P

EW

BM

BC

R R R R R R R R R R R R R R R R R R R R R R R R T T T T T T T T T T T T T T T T

Op

tim

um

IBI

UC-SNARL

CSBP

USFS-R5

Lin's concordance correlations:UC-SNARL.6 x CSBP.8 = 0.875UC-SNARL.6 x USFS.R5.7 = 0.924CSBP.8 x USFS.R5.7 = 0.936

ConcordanceCorrelations:0.88-0.94 IBIs0.84-0.88 O/Es

Reference24

Test16

<0.2 Rd xings/km and/or<25% Bank erosion

with no pollution source NO

YES

Metric screening and selection>optimal IBI developed

RIVPACS (site groups, taxa associations, etc)>optimal O/E models developed

CSBP SNARL USFS

2. Concurrent collection of field data and laboratory processing

Community similarityMetric similarity & precisionEndpoint similarity & precisionAssessment agreement

CSBP-IBI CSBP-O/E

SNARL-IBI

USFS-IBI

SNARL-O/E

USFS-O/E

cross-comparisons

1. Reference Selection:

3. Analysis

Reference24

Test16

<0.2 Rd xings/km and/or<25% Bank erosion

with no pollution source NO

YES

Metric screening and selection>optimal IBI developed

RIVPACS (site groups, taxa associations, etc)>optimal O/E models developed

CSBP SNARL USFS

2. Concurrent collection of field data and laboratory processing

Community similarityMetric similarity & precisionEndpoint similarity & precisionAssessment agreement

CSBP-IBI CSBP-O/E

SNARL-IBI

USFS-IBI

SNARL-O/E

USFS-O/E

cross-comparisons

1. Reference Selection:

3. Analysis

Standardizing to a common metric set further improves concordance among methods

Flow Chart of Study

0 0.1 0.2 0.3 0.4 0.5

between-methodafter resample to

500 count

between methodSNARL x USFS.R5

within-SNARL Bray-Curtis target

More similar < Bray-Curtis distance > Less similar

0

25

50

75

100

SS

P TF W TC

CW EC

RH

CN W F

OS

WC L

SV T

WP C

TP W D M CT

SL K

WF R

TB B

OT

O4

RB

OA

OX

OP

OB P

EW

BM

BC

Stream Site Code

UC

-SN

AR

L r

anke

d IB

Is

unimpaired

impaired

remove low 4 of 24 references = type I error = 17%

Example of assessed impairment: minimize test-reference overlap at equal type I error rates and compare sites judged as impaired vs unimpaired

Similar assessments were found between methods