XMM-Newton SSC: Pipeline Processing


Until March 2012, the XMM-Newton SSC performed routine processing of all XMM pointed (and slew) observation data on behalf of ESA using a dedicated Pipeline Processing System (PPS). In early 2012 the pipeline software and configuration details at that point were transferred to the XMM-Newton Science Operations Centre (SOC) at ESAC. From 1st March 2012, the SOC took over responsibility for routine processing.

While the pipeline processing task was transferred to the SOC, the SSC retained responsibility for the development of the cat9.0 pipeline that was to be used for the bulk reprocessing of all XMM-Newton data in December 2012, which would form the input to the 3XMM-DR4 X-ray source catalogue. Following the bulk reprocessing, the cat9.0 pipeline was transferred to the SOC during early 2013 and became the default processing pipeline at the SOC following the release of the 3XMM-DR4 catalogue on 23 July 2013.

The processing pipeline comprises a number of perl modules that perform a range of tasks on XMM-Newton data, acting as wrappers to a number of SAS tasks that perform most of the detailed science data reduction. The pipeline software and module codes are used to determine the data flows, provide some of the filtering logic and augment products, while the entire system is managed via an SQL database. The pipeline data flow and modules are shown in this schematic of the pipeline flow, while the modules from the cat9.0 pipeline can be accessed here.

A new pipeline has been inaugurated whenever a new SAS version has been incorporated or significant pipeline structure/functionality changes were made. Each pipeline is defined by a new label, configuration and release note. The tables below provide a résumé of current and past SSC pipelines.

When important software and/or calibration changes have accumulated, all available XMM ODFs are bulk reprocessed through a stable pipeline and static calibration data to re-impose processing/calibration homogeneity and to build XMM X-ray source catalogues (e.g. 3XMM-DR4).

Current Pipelines

Configuration ID   Started Processing Full Pipeline Release Note Summary Pipeline Release Note
04_cat9.0_20121220.153800 Documentation, Modules 2012-12-20 det_04_cat9.0_20121220.153800 sum_04_cat9.0_20121220.153800

Pipeline Configuration History

Configuration ID Started Processing Full Pipeline Release Note Summary Pipeline Release Note
13_prod8.4_20120118.0955002012-01-18 det_13_prod8.4_20120118.095500 sum_13_prod8.4_20120118.095500
13_slew8.4_20120118.1147002012-01-18 det_13_slew8.4_20120118.114700 sum_13_slew8.4_20120118.114700
12_prod8.4_20111111.1006002011-11-14 det_12_prod8.4_20111111.100600 sum_12_prod8.4_20111111.100600
01_slew8.3_20100920.1205002010-11-08 det_01_slew8.3_20100920.120500 sum_01_slew8.3_20100920.120500
01_prod8.3_20100920.1205002010-11-08 det_01_prod8.3_20100920.120500 sum_01_prod8.3_20100920.120500
07000001_20090303.1444002009-04-15 det_07000001_20090303.144400 sum_07000001_20090303.144400
02001005_20070531.1119082007-05-31 02001005_20070531.111908.release 02001005_20070531.111908.summary
(bulk reprocessing for 2XMM)
2006-10-26 06000520_20070308.163249.release 06000520_20070308.163249.summary
04000011_20040406.152514 2004-04-06 det_04000011_20040406.152514 sum_04000011_20040406.152514
02000051_20030917.155521 2003-09-17 det_02000051_20030917.155521 sum_02000051_20030917.155521
04000013_20030722.143236 2003-07-22 det_04000013_20030722.143236 sum_04000013_20030722.143236
02000050_20021220.132037 2002-12-20 det_02000050_20021220.132037 sum_02000050_20021220.132037
01000504_20021220.130209 2002-12-20 det_01000504_20021220.130209 sum_01000504_20021220.130209
02000038_20011207.102945 2001-12-07 det_02000038_20011207.102945 sum_02000038_20011207.102945
04000006_20010907.155438 2001-09-07 det_04000006_20010907.155438 sum_04000006_20010907.155438
02000037_20010806.101411 2001-08-06 det_02000037_20010806.101411 sum_02000037_20010806.101411
04000005_20010712.130904 2001-07-12 det_04000005_20010712.130904 sum_04000005_20010712.130904
02000035_20010618.143655 2001-06-18 det_02000035_20010618.143655 sum_02000035_20010618.143655
02000033_20010426.174813 2001-04-26 det_02000033_20010426.174813 sum_02000033_20010426.174813
02000032_20010212.141403 2001-02-12 det_02000032_20010212.141403 sum_02000032_20010212.141403
02000028_20010122.100409 2001-01-22 det_02000028_20010122.100409 sum_02000028_20010122.100409