We have uncovered an error in the 2XMM and 2XMMi catalogues which needs to be brought to the attention of users of these catalogues.
During the late stages in the preparation of the 2XMM catalogue in 2007, it was found that the EP_4_FLUX values were wrong due to a software bug in the srcmatch SAS task. The EP_4_FLUX value is the all-EPIC band-4 flux and is calculated as a weighted average of the band-4 fluxes from the available, individual cameras. The problem arose from the erroneous use of the PN band 1 flux value rather than the PN band 4 flux value when calculating this weighted average.
The problem was uncovered after the re-processing of all the available XMM-Newton data in 2006/7 had been completed. To avoid a further lengthy re-processing cycle to generate correct source lists, a correction was applied via an external script, run on all the existing observation summary source lists. These observation summary source lists are inputs to the software used to generate the final source catalogues.
While validation of a sample of source lists appeared to show the corrections had been successfully applied, a modest fraction (~ 17-19%) appear not to have been corrected. As none of these cases were in the validation sample, the problem was missed.
All observations that were re-processed in 2006/7 are potentially affected, but we have found that only observations between XMM-Newton revolutions 0369 (2001 Dec 13) and 1212 (2006 July 23) (and only a modest subset of those) are affected. Observations from revolutions beyond 1338 have been processed with a pipeline in which the srcmatch task bug had been fixed and are not affected in any way.
We also note that a new version of the catalogue, 2XMMi-DR3, which also contains a further ~64000 additional detections compared to 2XMMi, is planned to be released in late April 2010. This new version of the catalogue contains the correct EP_4_FLUX and SC_EP_4_FLUX columns.
However, the observation summary source lists that were delivered to the XMM-Newton Science Archive (XSA) in 2007 remain uncorrected. This means there will be discrepancies between the EP_4_FLUX column values in the new 2XMMi-DR3 catalogue and the observation summary source lists in the XSA and other sites that hold them. The SC_EP_4_FLUX values are specific additions to the catalogues and, as such, are not present in the observation summary source lists.
Figure 1 below summarises the overall statistical effect of this error in terms of both the flux ratio and flux difference normalised to the flux error. Key points are that the corrected fluxes are systematically higher than the incorrect values, but that in the majority of cases the difference is less than a factor of a few, typically amounting only to a few sigma shift. Nevertheless as the error produces values which are completely wrong and because it introduces a systematic shift to the fluxes, the effects of this error on data analysis projects based on the 2XMM or 2XMMi catalogues can be quite significant.
Left: Smoothed density plot of EP_4_FLUX ratio (corrected/uncorrected) as a function of EP_8_FLUX for detections affected by the coding error. The background density plot is on a linear scale; the contours enclose [99, 97, 90, 70, 50 & 5%] of the points. The mean ratio is ≈ 1.75; 90% of the points lie between ratios of ≈ 0.7 & 3.4.
Right: Smoothed density plot of EP_4_FLUX difference (corrected-uncorrected) normalised by the corrected EP_4_FLUX_ERR value as a function of EP_8_FLUX for detections affected by the coding error. The background density plot is on a linear scale; the contours enclose [99, 97, 90, 70, 50 & 5%] of the points. The mean difference is ≈ 0.7 σ whilst 90% of the points lie between -3 and +3.4 σ.
New full (and shortly) slim versions of the 2XMM (v1.2) and 2XMMi (v1.1) catalogues are available.
However, for users who wish to check detections or sources affected by this problem and/or correct the values in versions of the 2XMM or 2XMMi catalogues they have previously downloaded, we provide auxiliary .fits and .csv files which contain the corrections. These consist of a small subset of the columns for all detections in each catalogue, i.e. they are provided for each detection regardless of whether the EP_4_FLUX or SC_EP_4_FLUX values were right or wrong in the original catalogues.
The reason that we provide separate files for the 2XMM and 2XMMi catalogues is that, although the correct value of the EP_4_FLUX column is unique to each separate detection, independent of which version of the catalogue it appears in, the correct value of the SC_EP_4_FLUX is catalogue-version-dependent. This is because the SC_EP_4_FLUX value is an average over all detections of the same unique source and thus can change between different catalogue versions as further observations and thus detections are added.
The 9 columns are:
DETID- identifies each detection
SRCID- identifies the unique source the detection belongs to
OBS_ID- Observation identifier
WRONG_EP_4_FLUX- Original (i.e. wrong) weighted EPIC band 4 flux (over all available instruments) for the detection
CORRECT_EP_4_FLUX- correct weighted EPIC band 4 flux (over all available instruments) for the detection
WRONG_SC_EP_4_FLUX- Original (wrong) weighted EPIC band 4 flux over all detections comprising a unique source
CORRECT_SC_EP_4_FLUX- correct weighted EPIC band 4 flux over all detections comprising a unique source
AFFECTED- set to TRUE if the EP_4_FLUX column requires correction. Otherwise it is set to FALSE. Note that this value is set to TRUE even if the change in the numerical value is very small.
SC_AFFECTED- set to TRUE if the SC_EP_4_FLUX column requires correction. Otherwise it is set to FALSE. Note that this value is set to TRUE even if the change in the numerical value is very small.
The files can be accessed from
|2XMMcat_v1.0_corrected_ep4_subcols.fits||FITS binary||8.7 MB||MD5: ec09c04f41a19627b5e80a46de68e2c6|
|2XMMcat_v1.0_corrected_ep4_subcols.csv.gz||CSV||4.3 MB||MD5: 8fa5eddde501c2060ff8003f2d4cea27|
|2XMMicat_v1.0_corrected_ep4_subcols.fits||FITS binary||10.2 MB||MD5: 3e346b5cd62d7b66b314030bec14864c|
|2XMMicat_v1.0_corrected_ep4_subcols.csv.gz||CSV||4.9 MB||MD5: fbf509ab4ad336546a712d7ff8c43d5f|
SQL CREATE statements are provided to load the CSV format files into relational database management systems.
A list of the observation IDs (and corresponding revolution numbers) which contain catalogue detections affected by this error is available in Table 1.