Validate an LSST DM processCcd.py output repository against a set of LSST Science Requirements Document Key Performance Metrics. Also assess expected analytic models for photometric and astrometric performance following the LSST Overview paper.
setup validate_drp
validateDrp.py CFHT/output
will produce output, plots, JSON files analyzing the processed data in CFHT/output
. Metrics will be separately calculated for each filter represented in the repository.
Replace CFHT/output
with your favorite processed data repository and you will get reasonable output.
One can run validateDrp.py
in any of the following modes:
- use no configuration file (as above)
- pass a configuration file with just validation parameters (brightSnr, number of expected matches, ...) but no dataId specifications
- pass a configuration file that specifies validation parameters and the dataIds to process. See examples below for use with a
--configFile
This package also includes examples that run processCcd task on some CFHT data and DECam data and validate the astrometric and photometric repeatability of the results.
Pre-requisites: install and declare the following
pipe_base
from the LSST DM stack (note thatpipe_base
is included withlsst_apps
, which is the usual thing to install)obs_decam
from https://github.com/lsst/obs_decamobs_cfht
from https://github.com/lsst/obs_cfhtvalidation_data_cfht
from https://github.com/lsst/validation_data_cfhtvalidation_data_decam
from https://github.com/lsst/validation_data_decam
The obs_decam
, obs_cfht
, validation_data_cfht
, validation_data_decam
, validate_drp
products are also buildable by the standard LSST DM stack tools: lsstsw
or eups distrib
. But they (intentionally) aren't in the dependency tree of lsst_apps
. If you have a stack already installed with lsst_apps
, you can install these in the same manner. E.g.,
eups distrib install obs_decam obs_cfht validation_data_decam validation_data_cfht validate_drp
XOR
rebuild -u obs_decam obs_cfht validation_data_decam validation_data_cfht validate_drp
To setup for a run with CFHT:
setup pipe_tasks
setup obs_cfht
setup validation_data_cfht
setup validate_drp
As usual, if any of these packages are not declared current you will also need to specify a version or tag.
validation_data_cfht
contains both the test CFHT data and selected SDSS reference catalogs in astrometry.net format.
Run the measurement algorithm processing and astrometry test with
$VALIDATE_DRP_DIR/examples/runCfhtTest.sh
This will create a repository in your current working directory called CFHT.
The last line of the output will give the median astrometric scatter (in milliarcseconds) for stars with mag < 21.
To setup for a run with DECam:
setup pipe_tasks
setup obs_decam
setup validation_data_decam
setup validate_drp
As usual, if any of these packages are not declared current you will also need to specify a version or tag.
validation_data_decam
contains both the test DECam data and selected SDSS reference catalogs in astrometry.net format.
Run the measurement algorithm processing and astrometry test with
$VALIDATE_DRP_DIR/examples/runDecamTest.sh
This will create a repository in your current working directory called DECam.
The last line of the output will give the median astrometric scatter (in milliarcseconds) for stars with mag < 21.
There are also "Quick" versions to run one CCD for quick debugging and verification that things are running properly: examples/runCfhtQuickTest.sh
and examples/runDecamQuickTest.sh
.
Multiple filters can be processed by specifying a filter name for each visit. See examples/DecamCosmos.yaml
for an example. In brief:
# Visit - filter are matched pairs.
visits: [176837, 176839, 176840, 176841, 176842, 176843, 176844, 176845, 176846,
177341, 177342, 177343, 177344, 177345, 177346,]
filter: ['z', 'z', 'z', 'z', 'z', 'z', 'z', 'z', 'z',
'r', 'r', 'r', 'r', 'r', 'r', ]
# ccd list is iterated through for each visit-filter pair
ccdnum: [10, 11, 12, 13, 14, 15, 16, 17, 18]
While examples/runCfhtTest.sh
and examples/runDecamTest.sh
respectively do all of the processing and validation analysis, below are some examples of running the processing/measurement steps individually. While these examples are from the CFHT validation example, analogous commands would work for DECam.
-
Make sure the astrometry.net environment variable is pointed to the right place for this validation set:
export ASTROMETRY_NET_DATA_DIR=${VALIDATION_DATA_CFHT_DIR}/astrometry_net_data
-
Ingest the files into the repository
mkdir -p CFHT/input echo lsst.obs.cfht.MegacamMapper > CFHT/input/_mapper ingestImages.py CFHT/input "${VALIDATION_DATA_CFHT_DIR}"/raw/*.fz --mode link
-
Create the
runCfht.list
file from the YAML configuration filemakeRunList.py "${VALIDATE_DRP_DIR}"/examples/runCfht.yaml > "${VALIDATE_DRP_DIR}"/examples/runCfht.list
Once these basic steps are completed, then you can run any of the following:
-
To process all CCDs with the standard AstrometryTask and 6 threads use newAstrometryConfig.py:
processCcd.py CFHT/input @examples/runCfht.list --configfile config/newAstrometryConfig.py --clobber-config -j 6 --output CFHT/output
-
To process all CCDs with the old ANetAstrometryTask and 6 threads:
processCcd.py CFHT/input @examples/runCfht.list --configfile config/anetAstrometryConfig.py --clobber-config -j 6 --output CFHT/output validateDrp.py CFHT/output --configFile examples/runCfht.yaml
-
To process one CCD with the new AstrometryTask:
processCcd.py CFHT/input --id visit=850587 ccd=21 --configfile config/newAstrometryConfig.py --clobber-config --output tempout
-
Or process one CCD with the ANetAstrometryTask:
processCcd.py CFHT/input --id visit=850587 ccd=21 --configfile config/anetAstrometryConfig.py --clobber-config --output tempout
-
Run the validation test
validateDrp.py CFHT/output --configFile examples/runCfht.yaml
Note that the example validation test selects several of the CCDs and will fail if you just pass it a repository with 1 visit or just 1 CCD.
- Will likely not successfully run on more than 500 catalogs per band due to memory limits and inefficiencies in the current matching approach.
- The astrometric and photometric error models are formally valid for individual images. However, they are being applied here to the results from the set of images, which is implicitly looking at some sort of mean performance. E.g., the expected astrometric uncertainty is intimately related to the seeing of the image. For collections of images where most have a similar seeing, these estimates are useful and reasonable. However, if the data set analyzed consisted of a set of images distributed across a wide range of seeing values, then the fits here have less direct meaning.
bin.src/validateDrp.py
: Analyze output data produced by processCcd.pyconfig/cfhtConfig.py
: empty config overrides for Cfht. Edit to easily include config parameters in the examples.config/decamConfig.py
: empty config overrides for Decam. Edit to easily include config parameters in the examples.examples/runCfhtTest.sh
: CFHT Run initialization, ingest, measurement, and astrometry validation.examples/runDecamTest.sh
: DECam Run initialization, ingest, measurement, and astrometry validation.examples/runExample.sh
: General example runner.examples/Cfht.yaml
: CFHT YAML file with visits, ccd, paramaters for validateDrp.examples/Decam.yaml
: DECam YAML file with visits, ccd, paramaters for validateDrp.examples/DecamCosmos.yaml
: DECam COSMOS YAML file with visits, ccd, paramaters for validateDrp.python/lsst/validate/drp/base.py
: base routines for the modulepython/lsst/validate/drp/calcSrd.py
: calculate metrics defined by the LSST SRC.python/lsst/validate/drp/check.py
: coordination and calculation routines.python/lsst/validate/drp/io.py
: JSON input and output routines.python/lsst/validate/drp/plot.py
: plotting routinespython/lsst/validate/drp/print.py
: printing routinespython/lsst/validate/drp/srdSpec.py
: pipeBase Struct with SRD specifications. Access routine helper.python/lsst/validate/drp/util.py
: utility routinesREADME.md
: THIS FILE. Guide and examples.