Flagging: When Good Data Go Bad

Scott Schnee & Amy KimballNov 9, 2011

Interferometry Discussion Group

Initial Flagging

• Shadowing• Pointing Errors– Reported– Unreported

• Observing Log• Other obvious problems

Initial Flagging

• Shadowing– Issue at low elevations– Issue for compact arrays

In CASA for an ALMA data set:flagdata(vis=‘vis.ms’, mode=‘shadow’, diameter=12.0)

In MIRIAD for a CARMA data set:csflag(vis=‘vis.ms’, carma=true)

Initial Flagging

• Pointing Errors– Reported (e.g. wind)– Unreported (e.g. pointing model)

In MIRIAD for a CARMA data set:uvflag(vis=‘vis.ms’, select=“pointing(5,100000)”, flagval=flag)uvflag(vis=‘vis.ms’, select=“el(85,90)”, flagval=flag)

Initial Flagging

• Observing Log

Many observatories will note problems that affect the system for part or all of a track. This could be weather-related or hardware-related, and often requires that some data be flagged.

Initial Flagging

• Other obvious problemsSource Sys Temps (K) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15URANUS 235 216 1443 327 369 290 150 182 195 165 158 188 184 980 2043C84 183 163 1477 257 308 225 103 130 136 110 107 131 126 880 1203C111 187 167 1106 263 314 231 100 133 141 113 110 135 129 892 125L1451MM 183 165 1675 261 314 224 105 132 138 118 107 134 128 883 131L1451MM 184 164 1445 259 310 226 103 131 137 110 108 133 127 887 1280336+323 185 164 1327 258 311 226 107 132 139 112 9820 133 127 887 1223C111 185 165 1231 262 312 230 102 133 139 113 9449 134 129 884 1353C111 184 164 1320 258 311 225 103 131 136 114 9979 133 126 890 131L1451MM 181 162 1412 258 309 225 107 130 135 112 9928 131 126 876 121L1451MM 182 163 1318 257 308 226 106 130 136 110 105 131 126 892 1180336+323 182 163 1406 257 309 225 103 130 135 112 105 131 125 878 1213C111 183 163 1506 259 309 226 106 132 136 115 106 132 127 895 114

Initial Flagging• Other obvious problems

Tsys plots from NGC 3256 ALMA CASA Guide

What to look for?

• Plots of amplitude and phase vs time and channel• Iterate over– Antenna– Spectral window– Source

• Make plots of bandpass and gain calibrators first– Easy to find bad data of a bright point source– Hard to find bad data of a faint extended source

What to look for?

• Smoothly varying phases and amplitudes can be calibrated

• Discontinuities can not be calibrated• Features in the calibrators that may not be in the

target data can cause problems

Amplitude vs TimeFrom TW Hydra ALMA GuideColor: PolarizationOne spectral window (spw) plotted

Locating the Bad Data in plotms

Draw a box around the suspected bad data.

Locating the Bad Data in plotms

Click locate and CASA will send information about thedata to the logger.

Locating the Bad Data in plotms

Bad data can be flagged by pressing this button or using the flagdata tast at the CASA prompt.

Locating the Bad Data in plotms

Flagger’s remorse can be corrected by unflagging good data

From TW Hydra CASA GuideBrown and Green show phase calibratorsOrange shows TW Hya

Amplitude vs FrequencyBirdies

Amplitude vs FrequencyEdge Channels

Amplitude vs FrequencyEdge Channels

Data that should be flagged

Amplitude vs FrequencySpectral Lines in Bandpass Calibrator

From TW Hydra Band 7 GuideSpectral line in Titan

Phase vs TimePhase Jumps

First batch of data

Second batch of data

From Antennae ALMA CASA Guide

Possible Flagging Technique

1. Flag obviously bad data2. Calibrate the data3. Flag newly found bad data4. Re-calibrate5. Iterate (3, 4) or declare victory

After Calibration, Look Again

From NGC 3256 ALMA CASA guideAmplitude vs Time, after calibration

Sage Advice

From Rick Perley to a much younger Scott Schnee:“When in doubt, throw it out.”

Online flags• (from Steve Myers and Josh Marvil)

• Tbuff (before August 2011)

• Quacking sometimes necessary

Known RFI at science.nrao.edu• http://www.gb.nrao.edu/IPG/rfiarchivepage.html• https://science.nrao.edu/facilities/evla/observing/RFI/index

A flagging/calibration recipe

• EXAMINE bandpass/flux calibrator(s)• FLAG bandpass/flux calibrators• APPLY bandpass/flux calibration to itself

• APPLY bandpass/flux cal to phase cal sources

• EXAMINE phase cal sources• FLAG phase cal sources• APPLY phase calibration to itself

• APPLY bandpass/flux/phase cal to targets

• EXAMINE targets• FLAG targets

Iterate

Iterate

Repeat as necessary

Flagversion control in CASA

• Beware of applycal!

Recognizing low-frequency RFI

• Average all times: RFI visible on the shortest baselines

One EVLA user’s recommendation: flag these frequencies on all baselines at all times

Individual timeranges can be bad