📚 Guides
Miscellaneous

Miscellaneous

How does the Pipeline determine which days to process?

The general philosophy of this pipeline is that "it populates output directories, with minimal internal queuing logic". This means it does not track which processes it has already run. If you point it to some input and output directories, it will produce any output that is not there and use the output directories as a source of truth, "which processes have already been run".

If it finds an output directory for a sensor-day combination (and selected retrieval algorithm and atmospheric profile model), it will not re-retrieve it. If you want to re-retrieve something because the inputs have changed (which should usually not happen), you have to delete the respective output directory or point the pipeline to a new output path.

Hot to interpret the preprocessing error flags in Proffast 2.X?

The error flags are added up in the following way:

  • errflag  = 0: no error
  • errflag += 0.000.000.001 or 0.000.000.002: solar elevation below 1 deg
  • errflag += 0.000.000.010 or 0.000.000.020: DC mean too low (below DCmin)
  • errflag += 0.000.000.100 or 0.000.000.200: DC variance too high (above DCvar)
  • errflag += 0.000.001.000 or 0.000.002.000: DC correction failed
  • errflag += 0.000.010.000 or 0.000.020.000: Not enough ifg points available on any side of the ifg for requested resolution
  • errflag += 0.000.100.000 or 0.000.200.000: Out-of-band artefacts are too strong
  • errflag += 0.001.000.000 or 0.002.000.000: Not enough points available for calculating phase spectrum
  • errflag += 0.010.000.000 or 0.020.000.000: Invalid spectral calibration
  • errflag += 0.100.000.000 or 0.200.000.000: Check consistency of fwd and bwd pair of spectra
  • errflag += 1.000.000.000 or 2.000.000.000: Phase error > 0.005

The numeric values are the same for errflag_CO, but some of the steps/checks are not performed on the CO band.

What are the Units in the output files

VariableUnit
gndPhPa
gndTK
latdegdegrees
londegdegrees
altimmeters
appSZAdegrees
azimuthdegrees
XH2Oppm
XAIRno unit (ratio)
XCO2ppm
XCH4ppm
XCOppm
XCH4_S5Pppm
H2Ono unit (number of molecules)
O2no unit (number of molecules)
CO2no unit (number of molecules)
CH4no unit (number of molecules)
COno unit (number of molecules)
CH4_S5Pno unit (number of molecules)

Preprocess 5 in Proffast 2.3 fails with "Warning: inconsistent dualifg!"

We observed that Preprocess 5 (when using Proffast 2.3) cannot handle our old interferograms. The preprocess stops with the error message "Warning: inconsistent dualifg!". However, we can still process the interferograms with Preprocess 4 (using Proffast 2.2).

The log of the preprocess for these days looks like this:

================= Task 0 ================
/home/moritz-makowski/Documents/pipelines/erp-new-pressure/data/containers/retrieval-container-angry-benz/prf/preprocess/preprocess5 preprocess5ma_161102.inp
Return code: 1

Output:
 Reading input file...
 Done!
 Number of raw measurements to be processed:        3131
 Reading file names
 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.2853
 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.891
 ...
 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.3016
 Done!
 Read OPUS parms:           1
 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.2853
 Warning:
 inconsistent dualifg!
 This is a critical error. Quiet run option selected: End Programm


Errors:
 inconsistent dualifg!
============================================
============================================
Automatic Test SuiteConfiguration