XMM-Newton SSC: Pipeline Processing
Introduction
The XMM SSC performs routine processing of all XMM pointed observation data on behalf of ESA using a dedicated Pipeline Processing System (PPS). This pipeline reduces data from each of the EPIC, RGS and OM science instruments on XMM-Newton. The resulting products are a mixture of files that are useful as inputs to further scientific analysis by the user and files that provide a first cursory view of the data.
The SSC seeks to process and deliver observation data sets to ESAC (European Space Astronomy Centre), ready for delivery to the PI and XSA, on timescales of less than one month (but typically less than 2 weeks) of receipt of the raw ODF data from ESAC.
This role involves several stages:
- Secure transfer of the ODF data sets from ESAC
- Processing through a custom built pipeline, based around the SAS
- Transfer and return of EPIC-detected source lists to the Archival Catalogue and Database Subsystem (ACDS) at the L'Observatoire astronomique de Strasbourg (OAS) where they are cross-correlated against archival catalogue data (http://amwdb.u-strasbg.fr/ssc/products/acds.html)
- Distribution of data products to several SSC institutes for manual data quality and problem screening and
- Delivery of the processed, screened data back to ESAC for ingestion into the XSA.
ODFs are received from ESAC at a rate of about 15 per week, on average, and are processed on a quasi-continuous basis.
The SSC pipeline comprises a suite of perl modules, controlled by a Pipeline Control & Management System (PCMS), which uses a Sun Grid engine to distribute multiple processes, in parallel, across as many as 16 processors. In this way, multiple stages of several observations can be processed at the same time for maximum efficiency. The PCMS monitors and controls progress through an SQL database.
Each pipeline module executes one or more SAS tasks and performs other key functions in augmenting data products with essential information. They also perform essential data filtering logic. Here is a schematic of the pipeline flow and the modules from the current production pipeline.
A new pipeline is inaugurated whenever a new SAS version is incorporated or significant pipeline structure/functionality changes are made. Each is defined by a new label, configuration and release note. The tables below provide a résumé of current and past SSC pipelines. Access is also provided to the pipeline modules in the current production pipeline.
When important software and/or calibration changes have accumulated, all available XMM ODFs are bulk reprocessed through a stable pipeline to re-impose processing/calibration homogeneity and to build XMM X-ray source catalogues (e.g. 2XMM).
Current Pipelines
| Configuration ID | Started Processing | Full Pipeline Release Note | Summary Pipeline Release Note | |
|---|---|---|---|---|
| 13_prod8.4_20120118.095500 | Documentation, Modules | 2012-01-18 | det_13_prod8.4_20120118.095500 | sum_13_prod8.4_20120118.095500 |
| 13_slew8.4_20120118.114700 | Documentation, Modules | 2012-01-18 | det_13_slew8.4_20120118.114700 | sum_13_slew8.4_20120118.114700 |