2 Prelaunch Assessment of the NG VCM.pptx

The Aerospace Corporation (Aerospace)

Prelaunch Assessment of the Northrop Grumman VIIRS Cloud Mask

Thomas Kopp, The Aerospace CorporationKeith Hutchison, Northrop GrummanAndrew Heidinger, NOAA/STARRichard Frey, University of Wisconsin

IGARSS25 July 2011

[email protected] Satellite Systems

Outline

• Definitions of VIIRS Cloud Mask (VCM) contents and validation conditions

• High level review of the VCM logic

• Global results with the pre-launch VCM without any tuning

• Quantitative Improvements Using the Northrop Grumman (NG) tuning tool

• Methods for evaluating individual granules during Intensive Cal/Val (ICV) of the VCM


VCM Contents

• The VCM itself determines one of four cloud cover conditions for each pixel– Confidently Cloudy– Probably Cloudy– Probably Clear– Confidently Clear

• All downstream EDR products, except for imagery, require the VCM as an input

• Downstream products will use either the confidently cloudy or confidently clear condition– The probably clear/cloudy cases account for pixels that are not completely

cloud covered but due either to the difficulty of the scene or partial clouds such as cumulus, are not sufficiently clear to reliably determine the conditions at the surface


VCM Performance Metrics

•Probability of Correct Typing (PCT)

− PCT = (1 - Binary Cloud Mask Error) = {1 – [(VCM = conf. clear) & (Truth = conf.

cloudy) OR ((VCM = conf. cloudy) & (Truth = conf. clear)]/[total #pixels in each

geographic class – PCPC]

•Cloud Leakage (CL)

− CL = [(VCM = conf. clear) & (Truth = conf. cloudy)]/total #pixels in the geographic

class

•False Alarm Rate (FA)

− FA = [(VCM = conf. cloudy) & (Truth = conf. clear)]/total #pixels in the geographic

class

•Fraction of Pixels Classified as Probably Clear/Cloudy (FPCPC)

– FPCPC = [(VCM = prob. clear) or (VCM = prob. cloudy)]/total #pixels in the

geographic class


Overview of VCM Approach

• There are five possible processing paths in the VCM algorithm for the analysis of SDR data collected in daytime conditions

Cloud Spectral Tests Used to Determine Cloud Confidence

Water Land Desert Coast Snow

1. M9 (1.38 µm) Reflectance Test X X X (if TPW > 0.25

cm)

X X

2. M15-M16 (10.76 – 12.01 µm) Brightness Temperature Difference (BTD)

X X X X

3. Tri-Spectral M14, M15, M16 (8.55, 10.76, 12.01 µm) BTD Test

X

4. M15-M12 (10.76-3.70 µm) BTD Test

X(if no sun

glint)

X(if TOC NDVI >

0.2)

X(if Lat > 60 or < -

60)

X(if no sun glint

and if TOC NDVI > 0.2)

X

5. M12-M13 (3.70-4.05 µm) BTD Test

X (if –60 < Lat < 60) and no sun glint

X(if –60 < Lat < 60) and TOC

NDVI > 0.2

X(if –60 < Lat < 60)

6. M1 (0.412 µm) Reflectance Test X(if –60 < Lat <

60) 7. M5 (0.672 µm),

M1 (0.412 µm) Reflectance TestsX

(M5 if TOC NDVI ≥ 0.2;

M1 otherwise)

X(M5 if TOC NDVI ≥ 0.2;

M1 otherwise)

8. M7 (0.865 µm) Reflectance Test X

9. M7/M5 (0.865 / 0.672 µm) Reflectance Ratio Test

X X(if RefM5 ≥

LD_M5_Gemi Thresh)

Cloud Spatial Tests Used to Modify the Final Cloud Confidence Classification

Water Land Desert Coast Snow

10. I5 (11.45 µm) Spatial Test X

11. I2 (0.865 µm) Reflectance Test X


Pre-launch, Pre-tuned Global VCM Results

•VCM version used 1.5.0.48 from 2009

•The initial thresholds were used, the VCM for this testing was not tuned

•Comparisons made with collocated MOD35 C6 cloud mask and CALIOP matchups for comparison

– Cloudy for the VCM in this case included probably cloudy pixels– Clear for the VCM in this case included probably clear pixels

• Compared only 1-km CALIOP segments with either 0% or 100% cloud cover– Resulted in approximately 15 million collocations per month


Caveats and Notes to Global VCM Results

•Results intended to show “where we were” in late 2009

•Neither of the two I-band tests could be simulated using the proxy data, a significant source of error that will not be quantified until the post-launch validation of the VCM

•Thin cirrus has a major impact on the results

•Analysis limited to near-nadir views (MODIS viewing zenith angle of +/- 20 degrees)

•Hit rate = (# agree cloud + # agree clear) / total #

• Hanssen-Kuiper Skill Source (HKSS) = (# agree cloud * # agree clear) – (# disagree cloud * # disagree clear) /

(# agree clear + # disagree clear) * (# agree cloud + # disagree cloud)

•Results follow on the next few slides


VCM versus CALIOP, Global Results


VCM versus CALIOP, Polar and Non-Polar Results


VCM versus CALIOP, Day Results


VCM versus CALIOP, Night Results


VCM versus CALIOP, Land and Desert Results


VCM Designed to Exploit VIIRS 1.38-µm Data

MODIS vs VIIRS RSRs MODIS vs VIIRS TOA Radiances

VIIRS OOB Response is

orders of magnitude

less

MODIS OOBResponse is as large as the in-band response

Thin cirrus clouds will be more readily detected with VIIRS data than in MODIS


VCM Versus Heritage Performance, COT > 1.0

VCM and heritage performance are comparable when thin cirrus clouds are eliminated from the results


One Year Means of Hit Rates and Skill

July 2007 – June 2008, Comparison with CALIOP

Scene TypeMean VCM Hit

RateMean MOD35 C6

Hit RateMean VCM

HKSSMean MOD35 C6

HKSS

Global 79.0 86.9 62.0 72.51

60S-60N 84.1 89.9 74.0 78.1

Global day 83.4 89.0 66.1 76.3

Global night 75.1 85.1 59.6 69.1

60S-60N Water day

87.9 90.9 77.7 79.8

60S-60N Water night

83.2 90.5 76.4 76.0

60S-60N Land day

80.7 88.2 63.8 76.6

60S-60N Land night

80.1 87.4 65.0 74.7


Pre-Launch Tuning Approach

• Pre-launch tuning is based on 14 granules which employed Global Synthetic Data (GSD)– Of these 14, 11 contained land backgrounds

•These granules covered each VCM geographic type and ranged from straightforward to difficult scenes • GSD provides unique data to set the mid-point thresholds

– Typical methods of tuning, using on-orbit sensor data, rely upon 100% cloudy and 100% cloud free distributions

– GSD alone allows cloud distributions to be evaluated at the mid-point (50% cloudy) condition

•GSD allows setting thresholds and then minimize the distance between the confidently cloudy and confidently clear thresholds


Advantages Using Global Synthetic Data (GSD)

• GSD allows testing of the tuning process with proxy data and then apply the procedure to VIIRS-unique data– Tuning process is validated by :

–(1) tuning with GSD truth data developed with MODIS Relative Spectral Responses (RSR)

–(2) tested in the VCM using MODIS granules

–(3) quantitatively evaluated using manually-generated cloud data of the MODIS data

– Tuning for VIIRS data is then completed by examining changes in cloud distribution for each test in GSD truth data using the VIIRS RSR


Initial VCM Results Showed Following Needs

• Reduce the number of probably clear and probably cloudy (PCPC) classifications by adjusting the overall cloud confidence threshold

• Identify tests that generated the highest percentage of false alarms for each VCM background condition and tune the mid-point thresholds (i.e. 50% cloud cover condition) accordingly – only possible with GSD

• Further reduce the number of PCPC classifications, as necessary, by adjusting the distance between the mid-point thresholds of a given individual cloud test and the low and/or high threshold using cloud distributions in the GSD.


Initial Untuned VCM Performance - Land

Granule ID2001152_1600

2001196_1755

2001213_1210

2001213_1220

2002001_0340

2002032_1745

2002032_1750

2002118_0215

2003194_0905

2003299_1835

2003299_1840 Summary

land 628336 1127120 45030 54432 1367600 689449 1022770 673794 937571 1249880 108676

nPoorQual 0 0 0 0 0 0 0 0 0 0 0

nCldTruth 594640. 446056 14489 26133 382203 415431 393467 414653 88968 171321 28760

nClrTruth 33696 681069 30541 28299 985394 274018 629301 259141 848603 1078560 79916

nConfCldy 590182 466534 13811 32590 486446 458338 461027 539672 43910 241635 28341

nConfClr 6351 222995 3182 543 156780 49701 170838 65656 450710 352476 34965

nPrbCldy 2660 8692 1037 1032 54349 36563 23768 14685 3290 22254 1251

nPrbClr 29143 428904 27000 20267 670022 144847 367135 53781 439661 633517 44119

FalseAlarms 11974 46230 1316 7723 131286 82537 93687 125723 1378 89767 2630

Leakage 472 1346 28 38 1313 538 672 64 13895 1315 111

BinaryError 12446 47576 1344 7761 132599 83075 94359 125787 15273 91082 2741

FPCPC 0.05 0.39 0.62 0.39 0.53 0.26 0.38 0.10 0.47 0.52 0.42 0.38

pFalseAlarms 0.02 0.07 0.08 0.23 0.20 0.16 0.15 0.21 0.00 0.15 0.04 0.12

pLeakage 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.00

pBinaryError 0.02 0.07 0.08 0.23 0.21 0.16 0.15 0.21 0.03 0.15 0.04

PCT 0.98 0.93 0.92 0.77 0.79 0.84 0.85 0.79 0.97 0.85 0.96 0.87


Overview of the Pre-Launch Tuning Process

• Identify the tests causing the largest number of errors• Use GSD with MODIS RSRs to generate cloud cover

distributions for the cloud detection tests identified above– Generate distributions for 0%, 50%, and 100% cloud cover – Set key mid-point threshold using the 50% cloud cover, then

minimize low- and high thresholds

• Update VCM using these thresholds• Execute the updated algorithm on the set of MODIS

granules• Evaluate the performance using the manually generated

cloud masks• Assess the changes in performance


Example for a Case With Too Many PCPC Pixels

MODA.2001.196.1755 Manually-Generated Mask

Qthresh = 99%

Land - Pre

FPCPC 0.39pFalseAlarms 0.07pLeakage 0.00pBinaryError 0.07PCT 0.93

Qthresh = 90%

Land - Post

FPCPC 0.14pFalseAlarms 0.05pLeakage 0.01pBinaryError 0.06PCT 0.94


Specific Cloud Detection Case, GEMI Test (Land)

Changed from 1.95 to 1.87 Changed from 1.90 to 1.82 Changed from 1.85 to 1.78


Quantitative Impacts – GEMI Results

M7/M5 Land628336 1127120 45030 54432 1367600 689449 1022770 673794 937571 1249880 108676 SummarynPoorQual 0 0 0 0 0 0 0 0 0 0 0nCldTruth 594640 446056 14489 26133 382203 415431 393467 414653 88968 171321 28760nClrTruth 33696 681069 30541 28299 985394 274018 629301 259141 848603 1078560 79916nConfCldy 581516 408380 11854 20537 438229 404075 433527 466103 39542 211063 24899nConfClr 18824 650863 29320 21885 804817 183687 505091 134608 885731 960178 78081nPrbCldy 3268 8259 283 903 17190 23936 11252 12251 1490 11933 426nPrbClr 24728 59623 3573 11107 107361 77751 72898 60832 10808 66708 5270FalseAlarms 6113 11129 149 677 85209 55293 70980 53987 212 77525 984Leakage 2116 16834 1544 808 12483 15123 10395 298 39374 12827 2674BinaryError 8229 27963 1693 1485 97692 70416 81375 54285 39586 90352 3658FPCPC 0.04 0.06 0.09 0.22 0.09 0.15 0.08 0.11 0.01 0.06 0.05 0.07pFalseAlarms 0.01 0.01 0.00 0.01 0.06 0.08 0.07 0.08 0.00 0.06 0.01 0.05pLeakage 0.00 0.01 0.03 0.01 0.01 0.02 0.01 0.00 0.04 0.01 0.02 0.01pBinaryError 0.01 0.03 0.04 0.04 0.08 0.12 0.09 0.09 0.04 0.08 0.04 PCT 0.99 0.97 0.96 0.96 0.92 0.88 0.91 0.91 0.96 0.92 0.96 0.93

Previous untuned results

FPCPC 0.04 0.06 0.46 0.22 0.37 0.22 0.18 0.08 0.12 0.19 0.150.17

pFalseAlarms 0.01 0.01 0.01 0.01 0.08 0.11 0.08 0.19 0.00 0.06 0.010.06

pLeakage 0.00 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.04 0.00 0.010.01

pBinaryError 0.01 0.03 0.02 0.03 0.14 0.15 0.11 0.20 0.04 0.08 0.03

PCT 0.99 0.97 0.98 0.97 0.86 0.85 0.89 0.80 0.96 0.92 0.970.91


Tests Improved by the Pre-Launch Tuning Effort

• Reflectance test over desert (M1)• Reflectance test over land (M5)• Reflectance test over water (M7)• Ratio test over land (GEMI)• Ratio test over water (M7/M5)• Mid-Wave minus long wave infrared over snow (M12 – M15)• Mid-Wave infrared difference over snow (M12 – M13)


Performance After Daytime Tuning Completed

Performance Measure

Land Ocean Desert Snow

PCPC

SysSpec

38.3

15%

19.2

15%

14.7

15%

58.0

15%

False Alarms

SysSpec

12.4

7.0

6.0

5.0 10.5 22.5

Leakage (%)

SysSpec

0.5

3.0

0.3

1.0 2.3 0.02

PCT (%)

SysSpec

87.1

90.0

93.7

94.0 87.2 77.4

Performance Measure

Land Ocean Desert Snow

PCPC

SysSpec7.5

15%

22.5

15%

3.5

15%

5.1

15%

False Alarms

SysSpec4.6

7.0

1.4

5.0 2.1 3.7

Leakage (%)

SysSpec1.4

3.0

0.6

1.0 3.7 0.9

PCT (%)

SysSpec93.4

90.0

97.5

94.0 93.9 95.1

Untuned VCM: March 2010 Tuned VCM: November 2010


Tool for Visualization of the VCM

• The previous analyses reveal quantitative aspects of the VCM, but lack context

• Historically the capability to visualize the output from each individual cloud detection test has been used operationally at the Air Force Weather Agency

• Key to a useful visualization are two fundamental factors– It must overlay each test on applicable imagery– It must contain the reflectance/brightness temperatures used within

the cloud mask• This reveals if any bands have bad or saturated values

• The visualization should also note if any degraded conditions of note exist in the scene– These include aerosols, sun glint, and shadows

• The following pair of slides show this capability


Aerospace Visualization Tool – Example I


Aerospace Visualization Tool – Example 2


Conclusion

• Pre-launch validation of the VCM uses three different approaches to verify the VCM will meet expectations– Large scale quantitative analysis– Small scale quantitative analysis via GSD– Visualization of individual granules with each component cloud

detection test

• Results show promise that the VCM will meet or exceed its requirements

• Each of these methods will be employed in some form post-launch, though we will no longer need GSD as actual VIIERS data will be available

Education

2 Prelaunch Assessment of the NG VCM.pptx