Herschel Open Time Cycle 1 DP workshop

ESAC, 16-18 March 2011

- page 1

Spurs in HIFI dataSpurs in HIFI data

Colin Borys

- page 2

Artifacts in HIFI Data

• Generally, there are three major cleaning exercises with HIFI data.

1. Baseline subtraction2. Standing Wave removal3. Spur flagging

• In this talk/demo, we concentrate on spurious response in HIFI.

- page 3

Spur Definition and Detection

For the purposes of this discussion, we define spurs as those features which are generally Gaussian in shape and narrow (<50MHz). They can look like astronomical lines, and have the potential to ‘fool’ users.

- page 4

Related artifacts

Other artifacts due to saturated detectors or under pumped mixers can also corrupt HIFI spectra, but are much more obvious and easy to remove.

- page 5

Spur Detection in HIFI SPG

Because spurs can look like real lines, we search for them only in the loads. The algorithm used in the standard HIFI pipeline is as follows:

Loop through each WBS HTP at level 0.5 Loop through each cold load within this HTP Loop through each subband in the load dataset

1. Flag saturated pixels2. Compute second derivative of the load spectrum3. Mark points in the 2nd derivative that exceed a

threshold and fit as a spur.4. Flag the spur, and record parameters in a table (trend

analysis section of obs context).

As part of the level 0.5->level 1 pipeline, flux calibration is performed and as a result flags in the loads are propagated naturally to the science data.

- page 6

The obs context spur table

obs=getObservation("134281161” ,poolName="1342181161_sscan_dbs_1b_r4")

- page 7

The obs context spur table

obs=getObservation("134281161” ,poolName="1342181161_sscan_dbs_1b_r4")

- page 8

HSPOT and Spur Reporting

Based on repeated observations, we find that spur positions (LO vs IF) are generally stable and repeatable.

Weak spurs (in particular band 4b) can come and go based on the conditions of the instrument at the time of observation.

This means we warn users at the HSPOT planning stage if their observation is in danger of being corrupted by a spur.

- page 9

Diverse Spur Behavior

Strong spurs, such as the (now fixed) spur in 1a, can knock out an entire dataset.

Weak spurs can impact only a very narrow region (4b for example)

Some spurs have a predictable position in the IF as a function of the LO frequency.

- page 10

More spur pathology

1.Some spurs can move in IF over time. If an observation goes on a long time before a LO retune, the spur can ‘migrate’ out of the flagged region and complicate data reduction (most notably for spectral scans).

2.Spurs can be missed by the algorithm if the load is not as ‘smooth’ as expected.

Such data need to be flagged manually, and we discuss this in the demo.

- page 11

Spur impact on deconvolution

Historically, the spur finder was developed for use in spectral surveys, where artifacts such as these can significantly corrupt the result from the deconvolution. It has now matured into a full featured, automated pipeline module for ALL observations, regardless of mode.

Deconvolved output from data with an unflagged spur. The iterative nature of the algorithm propagates the spur over the entire spectral range.

Same spectrum, this time with the spur masked out. The result reveals some baseline issues that now need to be addressed.

Final result after simple median baseline subtraction.

Herschel Open Time Cycle 1 DP workshop

ESAC, 16-18 March 2011

- page 12

Flagging dataFlagging data

- page 13

Overview

• Whether it be spurs, overtly bad data, or some other reason, there are two main ways to flag data for exclusion

1. Channel level flags (via HIFI.MASK)2. Dataset level flags (via ROWFLAGS)

• Currently we do not have user friendly tools to let you flag data, but it is relatively straightforward via scripts, and one can (in some cases) use the SpectrumExplorer

• As with other scripts shown in this workshop, the general idea is to extract the data you want to manipulate, deal with it, then re-insert into the obs context.

- page 14

Tool dependent flag response

• Currently, HIPE tools respond differently to different flags.

• Deconvolution of SSCAN, for example, handles row flags and channel flags.

• RemoveFlaggedPixels is a task that lets users quickly remove flagged data via setting to NaN or interpolation

• HIPE help search on ‘rowflags’ will help you parse what flags correspond to.