Miscellaneous
How does the Pipeline determine which days to process?
The general philosophy of this pipeline is that “it populates output directories, with minimal internal queuing logic”. This means it does not track which processes it has already run. If you point it to some input and output directories, it will produce any output that is not there and use the output directories as a source of truth, “which processes have already been run”.
If it finds an output directory for a sensor-day combination (and selected retrieval algorithm and atmospheric profile model), it will not re-retrieve it. If you want to re-retrieve something because the inputs have changed (which should usually not happen), you have to delete the respective output directory or point the pipeline to a new output path.
How to interpret the preprocessing error flags in Proffast 2.X?
The error flags are added up in the following way:
errflag=0: no errorerrflag+=0.000.000.001or0.000.000.002: solar elevation below 1 degerrflag+=0.000.000.010or0.000.000.020: DC mean too low (belowDCmin)errflag+=0.000.000.100or0.000.000.200: DC variance too high (aboveDCvar)errflag+=0.000.001.000or0.000.002.000: DC correction failederrflag+=0.000.010.000or0.000.020.000: Not enough ifg points available on any side of the ifg for requested resolutionerrflag+=0.000.100.000or0.000.200.000: Out-of-band artefacts are too strongerrflag+=0.001.000.000or0.002.000.000: Not enough points available for calculating phase spectrumerrflag+=0.010.000.000or0.020.000.000: Invalid spectral calibrationerrflag+=0.100.000.000or0.200.000.000: Check consistency of fwd and bwd pair of spectraerrflag+=1.000.000.000or2.000.000.000: Phase error > 0.005
The numeric values are the same for errflag_CO, but some of the steps/checks are not performed on the CO band.
What are the Units in the output files
| Variable | Unit |
|---|---|
| gndP | hPa |
| gndT | K |
| latdeg | degrees |
| londeg | degrees |
| altim | meters |
| appSZA | degrees |
| azimuth | degrees |
| XH2O | ppm |
| XAIR | no unit (ratio) |
| XCO2 | ppm |
| XCH4 | ppm |
| XCO | ppm |
| XCH4_S5P | ppm |
| H2O | no unit (number of molecules) |
| O2 | no unit (number of molecules) |
| CO2 | no unit (number of molecules) |
| CH4 | no unit (number of molecules) |
| CO | no unit (number of molecules) |
| CH4_S5P | no unit (number of molecules) |
Preprocess 5 in Proffast 2.3 fails with “Warning: inconsistent dualifg!”
We observed that Preprocess 5 (when using Proffast 2.3) cannot handle our old interferograms. The preprocess stops with the error message “Warning: inconsistent dualifg!”. However, we can still process the interferograms with Preprocess 4 (using Proffast 2.2).
The log of the preprocess for these days looks like this:
================= Task 0 ================/home/moritz-makowski/Documents/pipelines/erp-new-pressure/data/containers/retrieval-container-angry-benz/prf/preprocess/preprocess5 preprocess5ma_161102.inpReturn code: 1
Output: Reading input file... Done! Number of raw measurements to be processed: 3131 Reading file names /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.2853 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.891 ... /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.3016 Done! Read OPUS parms: 1 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.2853 Warning: inconsistent dualifg! This is a critical error. Quiet run option selected: End Programm
Errors: inconsistent dualifg!========================================================================================Can I use the PyPy interpreter to make the Pipeline faster?
Maybe, but this is not tested. Since the pipeline spends almost the entire time on IO or running Fortran programs, using a faster Python interpreter will not have a measurable impact on the runtime.
What do the parameters in the OPUS files mean?
The list below was blatantly stolen from the TCCON wiki (https://tccon-wiki.caltech.edu/Main/I2SAndOPUSHeaders).
| Parameter ID | Description |
|---|---|
| Instrument Parameter | |
| DUR | Scan time [seconds] |
| PKA | Peak Amplitude |
| PKL | Peak Location |
| PRA | Backward Peak Amplitude |
| PRL | Backward Peak Location |
| HFL | High Folding Limit |
| LFL | Low Folding Limit |
| LWN | Laser Wavenumber |
| SSP | Sample Spacing Divisor |
| SSM | Sample Spacing Multiplicator |
| INS | Instrument Type |
| FOC | Focal Length |
| ASS | Number of Sample Scans |
| GFW | Number of Good FW Scans |
| GBW | Number of Good BW Scans |
| BFW | Number of Bad FW Scans |
| BBW | Number of Bad BW Scans |
| RSN | Running Sample Number |
| HFF | Digital Filter HFL |
| LFF | Digital Filter LFL |
| HUM | Humidity Interferometer |
| TLP | IFSSrcT |
| PIM | Pressure Interferometer |
| TSC | Scanner Temperature |
| SRN | Slice request number??? |
| Acquisition Parameter | |
| AQM | Acquisition Mode |
| SGN | Signal Gain, Sample |
| RES | Resolution |
| OPL | Optical path difference, long side [cm] |
| OPS | Optical path difference, short side [cm] |
| NSS | Sample Scans |
| FT Parameters | |
| APF | Apodization Function |
| PHR | Phase Resolution |
| SVF | SIV correction frequency |
| PCL | Phase Corr Operator Length |
| PCT | Phase Correction Thresh |
| ZFF | Zero Filling Factor |
| ZPL | ZPD Location |
| SCF | |
| DCL | |
| ZPA | |
| VER | Version Number |
| VDC | variability in the DC level |
| LST | laser sampling type [0: none; 1: InGaAs; 2: Si; 3: Dohe et al.; 4: other] |
| LSE | laser sampling error [the shift] |
| LSU | laser sampling uncertainty |
| Optic Parameters | |
| SRC | Source Setting |
| APT | Aperture Setting |
| FOV | Field of view in mrad |
| BMS | Beamsplitter Setting |
| VEL | Scanner Velocity |
| DTC | Detector Setting |
| HPF | High Pass Filter |
| LPF | Low Pass Filter |
| CHN | Measurement Channel |
| PGN | Preamplifier Gain |
| RDX | |
| SON | External Synchronisation |
| Sample Parameters | |
| SFM | Sample Form |
| LAT | Latitude |
| LON | Longitude |
| ALT | Altitude [m a.s.l.] |
| TOU | Average temperature [outside, °C] |
| POU | Average pressure [outside, hPa] |
| HOU | Average humidity [outside, %] |
| SIA | Average solar intensity |
| SIS | Standard deviation of solar intensity |
| WSA | Average wind speed [m/s] |
| WDA | Average wind direction |
| EXP | Experiment [filename] |
| Data Parameters | |
| DPF | Data Point Format |
| NPT | Number of Data Points |
| FXV | Frequency of First Point |
| LXV | Frequency of Last Point |
| CSF | Y - Scaling Factor |
| MXY | Y - Maximum |
| MNY | Y - Minimum |
| DXU | X Units |
| DAT | Date of Measurement |
| TIM | Time of Measurement |