Overview of the CASU pipeline processing

Please send any comments on this document to Mike Irwin.

Pipelines summary

• Summit pipeline: real time DQC and verified raw product to Cambridge. Proceeding on schedule with CASU software modules installed by JRL at JAC and tested on UFTI data under ORAC-DR.
• Standard pipeline: instrumental signature removal, catalogue production, astrometric and photometric calibration. Will run on a nightly basis in Cambridge, proceeding as planned apart from re-scheduling certain tasks because of delay in commissioning.
• Further processing pipeline: year 2 deliverable for PSF and galaxy profile fitting. Some aspects of this have been started early due to delays in commissioning.
• Advanced pipeline: database driven from WFAU Science Archive, deals with stacking and mosaicing, merging catalogues, list-driven photometry, difference imaging, overlap calibration. Again several aspects of this have been started earlier than originally planned.

Data Transfers

Raw data will appear from DAS in 32-bit integer NDF format. A copy of this raw data will be stored at JAC on LTO-I tapes. The summit pipeline will convert the NDF to single extension FITS using lossless Rice tile compression, which gives a factor of ~4 saving.

Each detector has its own independent DAS PC and summit data processing PC + LTO-I tape drives on each. The pipeline will write and verify the raw compressed single extension FITS files to 4 tapes in parallel for export to Cambridge, with an expected ~weekly duty cycle. Each tape holds 100Gb native i.e. equivalent to ~400 Gb of raw uncompressed FITS files.

In Cambridge the 4 tape streams will be ingested, verified and converted to Multi-Extension FITS (MEFs) and then stored (in compressed form) using online RAID arrays.

The raw data archive will be pushed to ESO in Garching via the internet.

Technical Challenges

• non-linearity measurement and calibration of arrays (at sector level?) [nearly all systems we have dealt with are non-linear]
• cross-talk between channels and detectors, potentially 128x128 matrix [several WFIs have cross-talk]
• image persistence from frame-to-frame, adjacency and temporal effects [two possible solutions developed but not used in anger yet]
• sky level variations on short timescales and/or spatially require several possible sky subtraction strategies [see later]
• rapid variablity of OH fringing requires specific defringing algorithms in addition to basic sky subtraction [a common problem in all imagers]
• seeing variations on short timescales [interleaving will lead to some interesting effects]
• possible time varying detector characteristics [impacts design of MSBs and flatfielding strategies]
• calibration and extinction monitoring [essential to get well-defined and regular monitoring - see later]
• data volume and throughput, particularly if much reprocessing is needed [lack of pipeline tuning lead-time potentially exacerbates this]
• generating robust Data Quality Control (DQC) measures [can only visually inspect a small subset of images]

Progress

• Interface Control Documents (ICDs) with JAC and WFAU agreed.
• FITS header definition (crucial for pipeline automation) finalised and test MEF data processed.
• Automated DQC monitoring tools developed using images and catalogues.
• Provisional pipeline progress monitoring tools with web interface.
• Programming completed for all basic modules.
• ORAC-DR test routines for UFTI delivered and installed.
• Software module development for standard and summit pipelines on target.
• Automated astrometric and photometric calibration developed.
• Practical method for dealing with exposure maps, noise variance weighting and bad pixel masks developed.
• V1 stacking and mosaicing and V1+2 difference imaging sofware completed.
• Series of simulation tests, and other reports and manuals finished and available on web page.
• Tookits/pipelines developed and run on optical data from:
• INT WFC (4x 2kx4k)
• ESO WFI (8x 2kx4k)
• KPNO & CTIO Mosaic1,2 (8x 2kx4k)
• AAT WFI (8x 2kx4k)
• Palomar WFI (6x 2kx4k)
• CFHT 12k imager (12 x2kx4k)
• and on NIR array data from:
• CIRSI mosaic on INT/DuPont (4x 1kx1k)
• INGRID on WHT (1kx1k)
• UFTI on UKIRT (1kx1k)
• ISAAC on VLT (1kx1k)

Astrometric Calibration

Expected WCS is ZPN with simple polynomial radial distortion of form
r'=k1.r + k3.r³
with possibly higher order r⁵ term needed.

Encoded in FITS using RA---ZPN CTYPE1 etc. keywords/values and using PV2_1, PV2_3 etc. to denote the radial polynomial coefficients.

Distortion is wavelength dependent with K > H > J e.g. at 0.4 deg. radius non-linear distortion is roughly 0.32% in J, 0.34% in H and 0.38% in K.

As an example: in K distortion at corners of detectors amounts to ~10 arcsec and leads to a differential distortion of a maximum of 0.2 arcsec when stacking detectors taken with an offset of 10,10 arcsec.

The scale distortion also leads to a photometric calibration effect since outer pixels project differently on sky compared to inner pixels. This leads with conventional processing, if uncorrected, to photometric systematics up to +/-1% over the array. These can be corrected for using the planned "mesostep" calibration squences, which will also correct for problems caused by scattered light or other non-uniform sky illumination in the flatfields.