Quality Control of Soil Moisture and TemperatureFor US Climate Reference Network

Basic Methodology

February 2009William Collins

USCRN

Complex Quality Control

The basic methodology is to use complex quality control—several largely independent tests are made, followed by data qualityand parameter value determination. The use of independent tests greatly increases the confidence inthe final determination of all sensor qualities as well as confidencein the best estimate of the parameter value.

Tests for soil moisture and temperature:

1)Range test for individual sensor values2)Step change test for individual sensor values3)Freezing test for soil moisture4)Persistence test, using last day’s values (mean, standard deviation,and largest difference used)5)Mean value test, comparing means of 3 sensors, using last day’svalues6)Inter-sensor test, using full values (can include mean value correction)7)Inter-sensor test, using step values8)Inter-level comparison of values9)Comparison of soil moisture with precipitation

Range Test for Individual Sensor Values

This test is performed independently of other tests to guaranteethat sensor values are within instrumental ranges. It is performed separately for each individual sensor value. This test also checks for missing data. Its results are passed on to the Complex Quality Control (CQC) algorithm via appropriate flags.

The following ranges are suggested:

Temperature: -30 to +55 C (used by OK Mesonet at 5 cm depth)

Moisture: 0.00 to 1.00 m3 water/m3 soil

Step Change Test for Individual Sensor Values

This test determines whether the change in parameter value from the previous value is physically reasonable. It is applied separately to each parameter value. Its results are passed on to the Complex Quality Control (CQC) algorithm via appropriate flags. The following 1-hour change values are preliminarily suggested:

Temperature: +/- 10 C

Moisture: +/- 0.50 m3 water/m3 soil

Freezing Test for Soil Moisture

If the soil is frozen, the moisture measurements are suspect. This test uses an estimate of the soil temperature at the same level to determine whether the soil is likely frozen or not. (It is to be determined later whether the soil temperature can go through CQC before soil moisture or not. That will determine whether the soil temperature used here is a preliminary estimate of its value or a qc’d value.) If the soil is estimated to be frozen, the flags for the soil moisture for all sensors at the same level are set appropriately.

Persistence Test

This test uses the parameter values from a full day. The OK Mesonet makes this test at the end of each day, but it could be done more frequently. For its use by the CQC algorithm, it is optimally performed using a day’s worth of data immediately preceding and including the present observation time.

From the day’s worth of data, a mean, standard deviation, and largest deviation from the mean are calculated for each sensor. If the largest deviation and standard deviation are below specified limits, the sensor is flagged appropriately. This determination is subsequently used by the CQC algorithm. The mean value may be used by later tests.

This test should be made sequentially, so that already qc’d data from previous times are used to qc the present time’s data. Two possible schemes seem reasonable: 1) perform the test daily, but use the previous day’s already qc’s data as well, or 2) perform the test at the frequency of the observations, using a day’s worth of previous (already qc’d) data.

Mean Value Test

This test directly uses the 3 means of co-located sensors as computed by the Persistence Test. It uses the same inter-sensor test procedure as the other Inter-sensor tests, to be explained on following slides. Appropriate flags are set for each of the 3 co-located sensors.

Inter-sensor Test, Using Full Values

A test, comparing 3 measurements for the same parameter, at the same location, was developed for use by USCRN for the 3 wires of the Geonor precipitation gauge. The same procedure can be used for the 3 moisture or temperature sensor measurements to determine if any is likely bad. The test computes the 3 inter-sensor differences and from them determines which, if any, is likely bad. A predetermined limit for inter-sensor differences is used in this test. By using more than one limit, it is possible to place confidence limits on the quality of the determination by this test. It is also possible to expand this test by making an estimate of sensor quality after first adjusting the values according to the difference in means from the last day’s worth of data, as found by the Persistence Test. Naturally, this can only be done if all 3 sensor means have good quality.

Inter-sensor Test, Using Full Values – Details

Procedure for determination of individual sensor quality:X is specified limit value.Assume all 3 measurements, s1, s2, s3 are bad.d12 = |s2-s1| d13 = |s3-s1| d23 = |s3-s2|

If ((d12<=X and d13<=X and d23<=X) or (d12<=X and d13<=X and d23>X) or (d12<=X and d13>X and d23>X) or (d12>X and d13<=X and d23>X) then s1 is good.

If ((d12<=X and d13<=X and d23<=X) or (d12<=X and d13>X and d23<=X) or (d12<=X and d13>X and d23>X) or (d12>X and d13>X and d23<=X) then s2 is good.

If ((d12<=X and d13<=X and d23<=X) or (d12>X and d13<=X and d23<=X) or (d12>X and d13<=X and d23>X) or (d12>X and d13>X and d23<=X) then s3 is good.

Inter-sensor Test, Using Step Values

This test is identical to the Inter-sensor Test, Using Full Values, with the exception that 1-hour changes to values are used instead of full values. Appropriate flag values are set, to be used by the CQC algorithm in determining a final value for the parameter (soil moisture or temperature) and the health of all sensors.

Inter-level Comparison of Values

The use on an inter-level comparison test will require a research and development effort. There are at least two possible methods. An optimal interpolation estimate of soil moisture or temperature could be made from measurements at other levels. A comparison of the OI estimate with the observations would determine an estimate of sensor quality and the appropriate flag would be assigned. An alternative, or possibly additional, method would use physics of soil moisture percolation and temperature wave changes in time to make estimates of the observed parameters. Such a methodology is likely beyond the scope of what should be pursued, partly because of its research and development needs, and partly because it would require data at several times and levels, and solar and precipitation data at the surface, for its implementation.

Comparison of Soil Moisture with Precipitation

This test compares the precipitation record with the soil moisture, particularly at 5 cm depth. Since a time lag is expected between any precipitation and soil moisture increase, a short time record of any precipitation is needed. Statistics need to be developed to establish reasonable time lags and a correspondence between the precipitation amount and the soil moisture increase, which likely will depend upon soil type (i.e. location). Appropriate data flags are set.

Complex Quality Control Algorithm

The CQC algorithm combines the complex of results from the various tests. It makes a final determination of the quality of each sensor and it specifies a best estimate to the value of moisture and temperature at each soil level. The development of the CQC algorithm has not yet begun. It is called “complex” both because it uses a complex of tests and because it may use complex logic. Certainly, the logic is non-linear in that the results of the various test interact non-linearly in the final decision regarding the health of the sensors. The estimate of the parameter value (moisture or temperature) will then depend on use of the good sensor values.