Flagging: When Good Data Go Bad
Scott Schnee & Amy KimballNov 9, 2011
Interferometry Discussion Group
Initial Flagging
• Shadowing• Pointing Errors– Reported– Unreported
• Observing Log• Other obvious problems
Initial Flagging
• Shadowing– Issue at low elevations– Issue for compact arrays
In CASA for an ALMA data set:flagdata(vis=‘vis.ms’, mode=‘shadow’, diameter=12.0)
In MIRIAD for a CARMA data set:csflag(vis=‘vis.ms’, carma=true)
Initial Flagging
• Pointing Errors– Reported (e.g. wind)– Unreported (e.g. pointing model)
In MIRIAD for a CARMA data set:uvflag(vis=‘vis.ms’, select=“pointing(5,100000)”, flagval=flag)uvflag(vis=‘vis.ms’, select=“el(85,90)”, flagval=flag)
Initial Flagging
• Observing Log
Many observatories will note problems that affect the system for part or all of a track. This could be weather-related or hardware-related, and often requires that some data be flagged.
Initial Flagging
• Other obvious problemsSource Sys Temps (K) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15URANUS 235 216 1443 327 369 290 150 182 195 165 158 188 184 980 2043C84 183 163 1477 257 308 225 103 130 136 110 107 131 126 880 1203C111 187 167 1106 263 314 231 100 133 141 113 110 135 129 892 125L1451MM 183 165 1675 261 314 224 105 132 138 118 107 134 128 883 131L1451MM 184 164 1445 259 310 226 103 131 137 110 108 133 127 887 1280336+323 185 164 1327 258 311 226 107 132 139 112 9820 133 127 887 1223C111 185 165 1231 262 312 230 102 133 139 113 9449 134 129 884 1353C111 184 164 1320 258 311 225 103 131 136 114 9979 133 126 890 131L1451MM 181 162 1412 258 309 225 107 130 135 112 9928 131 126 876 121L1451MM 182 163 1318 257 308 226 106 130 136 110 105 131 126 892 1180336+323 182 163 1406 257 309 225 103 130 135 112 105 131 125 878 1213C111 183 163 1506 259 309 226 106 132 136 115 106 132 127 895 114
Initial Flagging• Other obvious problems
Tsys plots from NGC 3256 ALMA CASA Guide
What to look for?
• Plots of amplitude and phase vs time and channel• Iterate over– Antenna– Spectral window– Source
• Make plots of bandpass and gain calibrators first– Easy to find bad data of a bright point source– Hard to find bad data of a faint extended source
What to look for?
• Smoothly varying phases and amplitudes can be calibrated
• Discontinuities can not be calibrated• Features in the calibrators that may not be in the
target data can cause problems
Amplitude vs TimeFrom TW Hydra ALMA GuideColor: PolarizationOne spectral window (spw) plotted
Locating the Bad Data in plotms
Draw a box around the suspected bad data.
Locating the Bad Data in plotms
Click locate and CASA will send information about thedata to the logger.
Locating the Bad Data in plotms
Bad data can be flagged by pressing this button or using the flagdata tast at the CASA prompt.
Locating the Bad Data in plotms
Flagger’s remorse can be corrected by unflagging good data
From TW Hydra CASA GuideBrown and Green show phase calibratorsOrange shows TW Hya
Amplitude vs FrequencyBirdies
Amplitude vs FrequencyEdge Channels
Amplitude vs FrequencyEdge Channels
Data that should be flagged
Amplitude vs FrequencySpectral Lines in Bandpass Calibrator
From TW Hydra Band 7 GuideSpectral line in Titan
Phase vs TimePhase Jumps
First batch of data
Second batch of data
From Antennae ALMA CASA Guide
Possible Flagging Technique
1. Flag obviously bad data2. Calibrate the data3. Flag newly found bad data4. Re-calibrate5. Iterate (3, 4) or declare victory
After Calibration, Look Again
From NGC 3256 ALMA CASA guideAmplitude vs Time, after calibration
Sage Advice
From Rick Perley to a much younger Scott Schnee:“When in doubt, throw it out.”
Online flags• (from Steve Myers and Josh Marvil)
• Tbuff (before August 2011)
• Quacking sometimes necessary
Known RFI at science.nrao.edu• http://www.gb.nrao.edu/IPG/rfiarchivepage.html• https://science.nrao.edu/facilities/evla/observing/RFI/index
A flagging/calibration recipe
• EXAMINE bandpass/flux calibrator(s)• FLAG bandpass/flux calibrators• APPLY bandpass/flux calibration to itself
• APPLY bandpass/flux cal to phase cal sources
• EXAMINE phase cal sources• FLAG phase cal sources• APPLY phase calibration to itself
• APPLY bandpass/flux/phase cal to targets
• EXAMINE targets• FLAG targets
Iterate
Iterate
Repeat as necessary
Flagversion control in CASA
• Beware of applycal!
Recognizing low-frequency RFI
• Average all times: RFI visible on the shortest baselines
One EVLA user’s recommendation: flag these frequencies on all baselines at all times
Individual timeranges can be bad