Miscellaneous

How does the Pipeline determine which days to process?

The general philosophy of this pipeline is that "it populates output directories, with minimal internal queuing logic". This means it does not track which processes it has already run. If you point it to some input and output directories, it will produce any output that is not there and use the output directories as a source of truth, "which processes have already been run".

If it finds an output directory for a sensor-day combination (and selected retrieval algorithm and atmospheric profile model), it will not re-retrieve it. If you want to re-retrieve something because the inputs have changed (which should usually not happen), you have to delete the respective output directory or point the pipeline to a new output path.

Hot to interpret the preprocessing error flags in Proffast 2.X?

The error flags are added up in the following way:

errflag = 0: no error
errflag += 0.000.000.001 or 0.000.000.002: solar elevation below 1 deg
errflag += 0.000.000.010 or 0.000.000.020: DC mean too low (below DCmin)
errflag += 0.000.000.100 or 0.000.000.200: DC variance too high (above DCvar)
errflag += 0.000.001.000 or 0.000.002.000: DC correction failed
errflag += 0.000.010.000 or 0.000.020.000: Not enough ifg points available on any side of the ifg for requested resolution
errflag += 0.000.100.000 or 0.000.200.000: Out-of-band artefacts are too strong
errflag += 0.001.000.000 or 0.002.000.000: Not enough points available for calculating phase spectrum
errflag += 0.010.000.000 or 0.020.000.000: Invalid spectral calibration
errflag += 0.100.000.000 or 0.200.000.000: Check consistency of fwd and bwd pair of spectra
errflag += 1.000.000.000 or 2.000.000.000: Phase error > 0.005

The numeric values are the same for errflag_CO, but some of the steps/checks are not performed on the CO band.

What are the Units in the output files

Variable	Unit
gndP	hPa
gndT	K
latdeg	degrees
londeg	degrees
altim	meters
appSZA	degrees
azimuth	degrees
XH2O	ppm
XAIR	no unit (ratio)
XCO2	ppm
XCH4	ppm
XCO	ppm
XCH4_S5P	ppm
H2O	no unit (number of molecules)
O2	no unit (number of molecules)
CO2	no unit (number of molecules)
CH4	no unit (number of molecules)
CO	no unit (number of molecules)
CH4_S5P	no unit (number of molecules)

Preprocess 5 in Proffast 2.3 fails with "Warning: inconsistent dualifg!"

We observed that Preprocess 5 (when using Proffast 2.3) cannot handle our old interferograms. The preprocess stops with the error message "Warning: inconsistent dualifg!". However, we can still process the interferograms with Preprocess 4 (using Proffast 2.2).

The log of the preprocess for these days looks like this:

================= Task 0 ================
/home/moritz-makowski/Documents/pipelines/erp-new-pressure/data/containers/retrieval-container-angry-benz/prf/preprocess/preprocess5 preprocess5ma_161102.inp
Return code: 1

Output:
 Reading input file...
 Done!
 Number of raw measurements to be processed:        3131
 Reading file names
 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.2853
 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.891
 ...
 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.3016
 Done!
 Read OPUS parms:           1
 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.2853
 Warning:
 inconsistent dualifg!
 This is a critical error. Quiet run option selected: End Programm


Errors:
 inconsistent dualifg!
============================================
============================================

Automatic Test Suite CLI