# intended for: observation and source screening for the serendipitous
# XMM Catalogues. Guideline and past experiences.
Select a number of sequence IDs (usually you will get 20 observations with every batch, so do them all together). Do Phase 1 screening on all of them first and select the observations suitable for Phase 2 screening. Make notes of any problems or questions and inform the main screener, who will let you know how to proceed in these cases.
Make notes of anything unusual. 'Unusual' means something you have not encountered before (see examples and the list of 'Troublesome ODFs Notes' on the internal web-pages of the pipeline processing at Leicester). If you are a beginner, talk with an experienced screener first and let him show you known problems (see also all the examples). Everything that you have not seen before, treat as unusual and discuss with the main screener.
It is important that you are consistent in your screening. If you are a beginner or have little experiences keep in close contact with the main screener and re-screen your first batch of observations, maybe even the second one. Study all examples given and make sure to have a check list of things to look out for at the beginning (e.g., what kinds or spurious sources are there? are real sources in areas where spurious sources are known to occur? look out for each in turn; don't forget too look for missed sources, etc; see example). Use Mike Denby's program 'addflags' if you want to add or remove flags to an observation that you have already 'finished' with DPSS.
In the following the details of Phase 1 and Phase 2 screening are explained, describing the
screening as it was done for the 1XMM Catalogue. In addition, a section with possible questions,
problems and answers is given. In the end, there is a section about the prospect to do faster
screening for future catalogues, which will mainly depend on a new source detection algorithm.
Use Netscape to view the web-pages for the sequence ID, that is, pages concerning the observation summary, EPIC processing summary and catalogue products. We do not look at OM and RGS products at all.
* Make sure all URLs are working
* OBS Summary page (check for EPIC products)
* EPIC Processing Summary page:
Make a note of which sequence ID/observation will go on to Phase 2 screening. Give the reason(s) if the observation will not be screened in Phase 2.
If there will be Phase 2 screening make a note for any camera if there is no exposure to be screened (give the reason, e.g., not active, no products, high background not screened) as well as which exposure will be screened in case there are multiple ones.
Make a note if a CCD is dimmer or brighter in both the exposure map and image (to make sure that you have checked it out).
Decide if the observation is suitable for the catalogue. Exclude observations of filamentary extended emission where a clumpy structure is indistinguishable from fore- or background sources and spurious sources. Exclude very bright extended sources where even bright real sources cannot be distinguished from the bright background (see examples). Note that at this stage only obviously unsuitable observations are excluded. This question will be re-addressed during/after Phase 2 screening (see below).
Make a note of problematic cases and/or questions and inform the main screener. He will then decide whether the exposure will be screened in Phase 2 or not.
Table 1.a gives the reasons for rejecting an observation for the catalogue (even if only one exposure is affected), while Table 1.b gives reasons for non-inclusion of an exposure. Any combination of the latter for all three cameras will lead to a rejection of the whole observation.
|Processing not completed||No web-pages available|
|Other problems during processing (see examples)||* Dim/bright CCD different in image and exposure
* Brighter outer CCDs in PN, probably corrupt data;
* Only central CCD in M1 or M2 observed or dimmer despite same GTIs (special observing mode);
* Central CCD in M2 shifted (special observing mode);
* Outside of FOV is brighter than the image;
* Different target but same observations as the one before;
|Target/observation too complex for screening||see examples|
|No products for that exposure||
* Instrument(s) not active; |
* No science settings of Mode or Filter;
* No exposures with duration >1000 sec
|No source list for that exposure||
* in PN: Small window mode, fast timing, fast burst |
* high background not screened
* empty image (no counts despite given good exposure time, see example)
For each observation/sequence ID that was selected during Phase 1 screening the purpose is:
|2||no visual inspection|
|3||the source lies within an extended region|
|4||there are nearby sources|
|5||the source position is suspect|
|6||the source is near an edge (of a CCD and/or the field-of-view)|
|7||a source comment is given|
'No visual inspection' flags:
Flag #2 is never set. It was useful only in the very beginning and might become so again, but for the time being ignore this flag.
'Nearby sources' flags:
Flag #4 is not set manually at all. However, the source comment 'Near bright source' (with flag #7 set) is common and indicates that a source lies in an area where spurious detection in the wings of a bright sources can occur.
'Position offset' flags:
Flag #5 is set when the position is clearly offset (see examples here) and when the source is split by a CCD gap or other (see examples here). Under certain circumstances a position could be affected by an edge of a CCD or the field of view (see examples here), but it is not always clear when a position is simply uncertain or when it is clearly offset. We would need the information of the uncertainty region as well but this would take too much time to find out. As a rule the flag is considered as a warning only and set rarely, with the source comment 'Position offset' strengthening the case.
'Near edge' flags:
Flag #6 should not be set manually except in special cases, like when a source is split by a CCD gap, or when a position is apparently offset due to a CCD edge or edge of the field of view (see also flag #5). The source detection and fitting algorithm is made to correct for counts missed in CCD gaps and at edges, though occasionally it fails (see the first example here). The edge of the field of view may have a different effect on the count rate and source position, and the flags might be useful in this case. However, the field of view is not always obvious and we therefore use this flag and the source comments 'Near edge of field' and 'Near edge of CCD' only when a source is bright and we suspect that it may be affected by such an edge.
'Source comment' flags:
Flag #7 indicates if a comment is given for this source. The source comment is given either in combination with another flag (to give an explanation why that flag has been set) or on its own as a warning that the source could be affected by its environment (see also Table 3 for its usage). Do not forget to set this flag when you give a comment!
Most extended sources are easily recognizable (see here to familiarize yourself with the diversity of extended sources), but there are cases where it is not so easy (see examples here). You can also compare with the composite image (if it is a useful one) to determine the existence and the extent of faint extended sources.
The importance of identifying extended sources is mainly due to the fact that often spurious detections are connected with them. On the other hand, since the source detection only assumes point sources, the background in areas of extended emission may not be well determined and the source counts of a source lying in such an area may be affected. We therefore attempt to flag all real sources within the visibly brighter background of extended emission with flag #3 (and a comment 'In extended' as well), while spurious detections usually come in patches where the background is fairly high or the gradient is steep. Only those patches will be flagged as false (except when a source in this area is obviously real, that is, all reasonably 'bright' sources). In these areas we will therefore 'miss' faint sources since we cannot distinguish them from the spurious detections. All we can do here is to make a statistical judgement: where many spurious detections are found this method catches all spurious detections while an occasional faint real source is also flagged as false, and in areas where no patches appear we keep all sources and accept that there is an occasional spurious detection among them.
Clumpiness in extended sources can also cause detections. These clumps are usually intrinsic to the extended source (e.g., clumps in a jet, but also a smooth narrow filament). It is not our intention to make distinctions here between 'real' clumps and the spurious detections that are caused by the general higher background (see example here), which can often be found close together. We therefore will flag all these detections as false but in case we think that a clump is a real part of the extended source we can use the source comment 'Part of extended' which indicates to the user that this detection is not just caused by the brighter background alone. However, one has to be careful to distinguish between clumpy parts of an extended emission and point sources within an extended emission, like stars in a galaxy or galaxies in a galaxy cluster (see here). The distinctions are not always clear and in case of doubt you should discuss this with the main screener.
Warm pixels are defined as being less bright than bright pixels, that is, the sum of all affected image pixels is ~20 counts for bright pixels and ~<15 for warm pixels. They are especially noticeable and frequent in M2. Do not attempt to flag these in M1 and M2 (mainly because there are too many). Use the DPSS-log report comment if you find one warm pixel in an exposure. Occasionally there is a 'warm' pixel in the PN, which in this case should be flagged as either 'Bright pixel' or 'Warm pixel'.
The DPSS-log report and the flagged source lists and other products created by the DPSS program will be delivered back to Leicester.
Make sure that the DPSS-log report uses only the given comments. That way it will be easier to automatically extract information, in particular the coordinates of new sources, at a later stage. If you have to make a comment that is new inform the main screener who will implement the new comment for all to use.
Classify the observation (based on experience) as to how easy/difficult it is to distinguish between spurious detections and real sources in problematic areas. Give an explanation. Difficult observations might be excluded from the catalogue at a later stage.
Write a list of observations/sequence IDs that you have screened with the following information (see
example): has Phase 2 screening been done or not? If not, give the
reason here. Which source lists (if ambiguous) have been screened? Note if a source list could not
be screened and why (e.g., high background not screened). Is the observation suitable for the
catalogue? If not, give a short explanation.
(To be done later at Leicester, by the main screener)
Phase 1 screening: check for questions and problems, maybe pass them on to people who might know the answers or should be informed of so far unknown problems. Make the final decision whether Phase 2 screening should be conducted.
Phase 2 screening: check each overlaid source list and their flags (with different colours for flag #1 and #3) for inconsistencies (especially with respect to the extent of extended sources and the knowledge of hot pixels/bad columns etc). Inform the screener if anything obvious was missed. See, for example, the composite images of the 1XMM catalogue before and after source-level screening here.
Screening log-files (see example): tabulate the observations looked at, reasons for rejection from the catalogue, screened exposures (if no exposure was screened for that particular observation state why not), and the classification of the observation (with regard to the quality of the discrimination of spurious and real detections, see Phase 2 screening). This is mainly for statistical purposes and reference later on.
Make sure that every up-date to the DPSS comment files and the source
comments are distributed to all screeners so that the comments used by all screeners are homogeneous.
Look also at the examples here.
Use a list of accepted observations and exposures for inclusion in the catalogue (also filtered by release date), as in the summary list of observations for the 1XMM catalogue, that is, there is no need to decide manually which observations and exposures are potentially useful for the catalogue.
Check the images, exposure maps, source overlays and flare backgrounds for the relevant exposures from the above mentioned list for any processing problems that may influence the source detection process (e.g., if the exposure map shows different GTIs than the image). See Table 1.a for a list of these reasons.
Make a decision based on the Netscape images whether the observation is not suitable at all for inclusion in the catalogue. Only the worst cases will be excluded at this stage. Uncertain cases will need to be decided upon later after Phase 2 screening, mainly because the Netscape images do not always have the best contrast set to make such a decision. Also, a decision at the end of the whole screening will be more consistent.
Classify the nature of the observation/target (e.g., bright central source with wings; extended source: faint or bright, smooth or clumpy; crowded field; etc).
Identify target (using the ACDS products)?
The observation/exposure itself will be flagged or classified according to whether spurious detections are found or not,
and how much the spurious detections affect the source finding (that is, how many spurious detections are there? How
large is the affected area? Is the affected area well defined?). The observations will also be classified
according to the nature of the target.
No individual source flagging is performed but areas where obvious spurious detections occur are
being marked and all sources within are flagged with a warning, regardless whether the particular
source is obviously real or not. In other words, it is left to the user to decide whether he wants to use such areas
and to flag the sources therein himself.
As before every exposure is being scanned and individual sources are being flagged. It will be
expected that only few detections are spurious. It would be easier to scan images and source
overlays with ds9 (better contrast, easy blinking between images with and without overlay and
between images from different cameras). Any source that will be flagged can be identified by a
region number. This will be used in a simple ASCII text to give the flags and comments. After an
exposure/observation is finished the comments in this ASCII file will be translated into the source
list (preferably as batch job). This way the long overheads in the DPSS (with its time consuming
access of the fits-tables) can be avoided.