Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
LCMAP Validation: An annual time-series land cover reference dataset
Methods and QA/QC results
Bruce Pengra, SGT, Contractor to the USGS
U.S. Geological Survey / EROS
November 7, 2018
USGS / USFS Collaboration
LCMAP / EROS (USGS)LCMS / Utah State University (USFS)
Reference Data needs:
• Characterize and validate the LCMAP annual land cover map
• Use with map layer to estimate of land cover and land cover
change
• Meet Forest Service / LCMS requirements (validation, model
training, etc.)
LCMAP Annual Land CoverCONUS/National Coverage
Landsat resolutionAnnual land cover 1985 to current
Ongoing annual updates
Conus LCMAP annual land cover simulated from NLCD data
LCMAP Annual Land CoverCONUS/National Coverage
Landsat resolutionAnnual land cover 1985 to current
Ongoing annual updates
Conus LCMAP annual land cover simulated from NLCD data
Reference Data Requirements
per LCMAP land cover validation
▪ Temporal coverage 85 to current
▪ Spatial coverage, CONUS▪ Landsat resolution (and grid)▪ Classified to match Land Cover
Trends legend▪ Can be updated annually ▪ Within LCMAP budget
▪ Delivered at the same time LCMAP products are
Methods for Meeting those Reference Data Requirements
➢ The Data & Tools
➢ The Response Design
➢ The Sample Design
➢ The QA/QC process
➢ Application
Base Sample of 25,000 plots
Temporal Coverage 1985 -current
Using sate l l i te data and/or aer ia l imagery for large area land cover accuracy assessment is wel l establ ished
Landsat time series data
1, 2, 3, 4, 5, 6, 7, 8, 9, 10
Google Earth high resolution imagery
Tools to make the process efficient
TimeSync, developed at OSU for US Forest Service.
Displays Landsat anniversary date image
Plots values through time series
Records and manages interpretations
The TimeSync Application manages Landsat data display and recording of interpretations for pre-defined sample locations
Plotted values for the sample pixel, showing values for all clear pixels
Anniversary date Landsat images, selected by the user from the usable images in each year
Google Earth
Efficient access to a time-series of high resolution images
FS = Forest Sample PlotLabel
GS = Wetland
Forest Service
Crosswalk
LCMAP Crosswalk
All Variables
Labeling Protocol
Land Use / Land Cover / Change
Methods for Meeting those Reference Data Requirements
➢ The Data & Tools
➢ The Response Design
➢ The Sample Design
➢ The QA/QC process
➢ Application
Methods for Meeting those Reference Data Requirements
➢ The Data & Tools
➢ The Response Design
➢ The Sample Design
➢ The QA/QC process
➢ Application
Joint Response DesignTodd Schroeder – USFS Warren Cohen – USFS/OSU
Interpretation Protocol • land use variables• land cover variable• land change processes
Joint Response Design
Land Use:(primary and secondary)
DevelopedForestAgricultureNon-forest WetlandRangeland/PastureOther
Land Cover:(primary and secondary)
TreesShrubsGrass/Forb/HerbImperviousBarrenSnow/Ice Water
Change Processes:FireHarvestMechanicalStructural DeclineSpectral DeclineWindHydrologyDebrisGrowth/RecoveryStableOther
Methods for Meeting those Reference Data Requirements
➢ The Data & Tools
➢ The Response Design
➢ The Sample Design
➢ The QA/QC process
➢ Application
Methods for Meeting those Reference Data Requirements
➢ The Data & Tools
➢ The Response Design
➢ The Sample Design
➢ The QA/QC process
➢ Application
The Sample Design
The Sample Design
The Sample Design
The Sample Design
The Sample Design
Methods for Meeting those Reference Data Requirements
➢ The Data & Tools
➢ The Response Design
➢ The Sample Design
➢ The QA/QC process
➢ Application
Methods for Meeting those Reference Data Requirements
➢ The Data & Tools
➢ The Response Design
➢ The Sample Design
➢ The QA/QC process
➢ Application
The QA/QC process
Validation of the Validation
The literature repeatedly points out that no perfect ground truth data exists with which to compare validation data 13,14,15,16,17,18,19
With no absolute “gold standard” available, Olofsson (2014)13 suggests that, “in practice, only assessing interpreter variability may be feasible.”
The QA/QC process
Our primary QA/QC activity is based on comparison of two independent interpretations (at ~60% of all plots)
• Agreement between these independent interpretations provides a measure of interpretation consistency that was used for quality control and for inferring the consistency of the full data set.
Assessment of the agreement (prior to review-and-revision) is calculated from a random sample of approximately 25% of all plots
• The assumption is that this is a baseline agreement number that will be improved by review and revision
The QA/QC process
Three goals:
1. Improve the interpreter consistency by providing feedback
2. Improve the data by correcting interpretation mistakes
3. Quantify the variability of the reference data – pre review
The QA/QC process
Following collection of each set of plots (~1000 plots each), several forms of feedback are provided to the interpreters
An individual contingency matrix showing where that interpreter had disagreed with interpretations by the other members of the interpretation team was provided.
Inte
rpre
ter
13
7
Interpreter 137 versus all other interpreters (set 10)
Set 10 Other Interpreters
Water Developed Disturbed Barren Forest Grass-shrub Ag Wetland Total Agreement %
Water 151 1 13 165 92
Developed 262 5 267 98
Disturbed 11 14 12 3 2 1 43 33
Barren 1 1 0
Forest 28 1706 8 97 1839 93
Grass-shrub 68 17 65 156 2204 96 94 2700 82
Ag 2 32 328 362 91
Wetland 1 33 64 98 65
Total 151 341 68 65 1907 2248 426 269 5475
4729agreement pixels
Agreement % 100 77 21 0 89 98 77 24 86.4overall agreement %
The QA/QC process
Following collection of each set of plots (~1000 plots each), several forms of feedback are provided to the interpreters
Review Notes
Plot 19497: Illogical call of primary use/cover Rangeland/Trees by 105. Tree line is less than 120 ft wide and therefore does not meet Forest criteria as 146 has. Use should be Other with dominant Tree cover. NHAP 1985 high resolution seems to show same (majority) tree line as later in time series.
Plot 19732: Both call GFH Rangeland but 109 includes a Fire in 2008 while 105 calls Stable. Review of NBR and chips does support a Fire in 2008 that was missed by 105.
Plot 20000: Illogical primary use/cover of Rangeland/Trees by 105.
Answers to Questions submitted by interpreters
The QA/QC process
Following collection of each set of plots (~1000 plots each), several forms of feedback are provided to the interpreters
Topical reviews of best practices and interpretation tips for classes of concern
The QA/QC process
All QA/QC duplicate plots (60% of all plots)
Overall between interpreter agreement for each of the first 22 sets
The QA/QC process
Some classes showed especially good improvement over time
The QA/QC process
Three goals:
1. Improve the interpreter consistency by providing feedback
2. Improve the data by correcting interpretation mistakes
3. Quantify the consistency of the reference data – pre review
Three goals:
1. Improve the interpreter consistency by providing feedback
2. Improve the data by correcting interpretation mistakes
3. Quantify the consistency of the reference data – pre review
The QA/QC process Improving the data by correcting interpretation mistakes
60% of all plots get a second interpretation
These plots are candidates for review based on a variety of agreement criteria• Overall agreement• Any disagreement related to change process• Belong to leading categories of disagreement
34% of duplicate plots are flagged for review by senior interpreters based on these criteria • Reviewers can identify the better interpretation of the two or suggest other revisions when it is clear an error was made• Instructions are recorded to revise the original interpretation where needed
Corrections are made to approximately 12% of all reference data plots to arrive at the final data• Corrections range from minor changes in a single year to major changes for all 33 years • They tend to be minor, such as recording a fire or harvest in the wrong year
The QA/QC process
Three goals:
1. Improve the interpreter consistency by providing feedback
2. Improve the data by correcting interpretation mistakes
3. Quantify the consistency of the reference data – pre review
The QA/QC process
Three goals:
1. Improve the interpreter consistency by providing feedback
2. Improve the data by correcting interpretation mistakes
3. Quantify the consistency of the reference data – pre review
The QA/QC process
The Random sampleBetween analyst agreement for duplicate interpretations at 5085 plots (x33 years) selected randomly for estimating initial agreement among interpreters over the full data set.
B Interpretations
Develope Disturbe Grass-Water d d Barren Forest shrub Ag Wetland Ice-Snow Total A agree
Water 8579 58 51 3 33 117 123 8964 95.71%
s Developed 76 6204 87 6 382 493 182 18 7448 83.30%
n Disturbed 15 81 839 33 375 189 65 113 1710 49.06%
atio
Barren 3 1 14 972 31 1032 1 2054 47.32%
et Forest 421 312 42130 2774 166 692 46495 90.61%
Inte
rpr
Grass-shrub 91 622 210 956 2425 55353 2467 380 62504 88.56%
A
Ag 46 292 94 14 182 2283 27740 87 30738 90.25%
Wetland 280 86 911 489 99 5975 7840 76.21%
Ice-Snow 33 33
147825
88.10%
100.00%
agreement pixels
overall agreement
Total 9090 7679 1693 1984 46469 62730 30720 7388 33 167786
B agree 94.38% 80.79% 49.56% 48.99% 90.66% 88.24% 90.30% 80.87% 100.00%
Methods for Meeting those Reference Data Requirements
➢ The Data & Tools
➢ The Response Design
➢ The Sample Design
➢ The QA/QC process
➢ Application
Application
➢ Validation and Characterization of LCMAP annual land cover
➢ Validation and characterization of LCMAP land cover change
➢ Estimation of national land cover composition
➢ Estimation of national land cover composition change
Application
Validation and Characterization of LCMAP annual land cover Comparison of the Map and Reference data land cover labels at each of the 25,000 sample points for each of 33 years
~825,000 points of comparison
Application
Estimation of national land cover composition & composition change
Class 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Water 1.7% 1.8% 1.8% 1.8% 1.8% 1.7% 1.7% 1.7% 1.7% 1.8% 1.8% 1.8% 1.8% 1.8% 1.9% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.9% 1.9%
Developed 4.2% 4.2% 4.2% 4.3% 4.3% 4.4% 4.4% 4.4% 4.5% 4.5% 4.6% 4.6% 4.7% 4.7% 4.8% 4.8% 4.9% 5.0% 5.1% 5.2% 5.3% 5.4% 5.5% 5.6% 5.5% 5.6% 5.6% 5.6% 5.7% 5.7% 5.8% 5.8% 5.8%
Disturbed 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
Barren 0.6% 0.7% 0.6% 0.6% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7%
Forest 29.7% 29.7% 29.6% 29.5% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.4% 29.4% 29.4% 29.4% 29.4% 29.3% 29.3% 29.2% 29.3% 29.4% 29.3% 29.4% 29.4% 29.3% 29.2% 29.1% 29.0%
GrassShrub 38.4% 38.5% 38.6% 38.7% 39.0% 39.1% 39.3% 39.4% 39.5% 39.4% 39.4% 39.4% 39.3% 39.3% 39.2% 39.2% 39.1% 39.1% 39.2% 39.2% 39.2% 39.1% 39.1% 39.2% 39.1% 39.0% 39.0% 38.8% 38.7% 38.7% 38.6% 38.7% 38.7%
Agriculture 20.2% 20.1% 20.0% 20.0% 19.9% 19.7% 19.5% 19.4% 19.3% 19.3% 19.2% 19.2% 19.2% 19.2% 19.2% 19.1% 19.0% 18.9% 18.7% 18.6% 18.6% 18.5% 18.5% 18.4% 18.4% 18.4% 18.4% 18.5% 18.7% 18.7% 18.8% 18.8% 18.9%
Wetland 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.0% 5.0% 5.1% 5.1% 5.0% 5.0% 5.0% 5.0% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1%
Ice&Snow 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
Application
Regional versions of the validation and area estimate applications