Describe the Enhancement

Theses refinement to the handling of missing data were discussed on July 12, 2023 at the TC-Diag project-wide meeting (see meeting notes). As of MET version 11.1.0, if TC-Diag tries to read data from a GRIB file that does not exist, or can't find an individual field of data in a file that does exist, it exits with bad status:

ERROR  : 
ERROR  : get_series_entry() -> Could not find data for TMP/P1000, ValidTime = 20220924_060000 in file list:
ERROR  : :/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/gfs/gfs.0p25.2022092400.f000.grib2,/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/gfs/gfs.0p25.2022092400.f006_BAD.grib2,/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/gfs/gfs.0p25.2022092400.f012.grib2,/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/gfs/gfs.0p25.2022092400.f018.grib2,/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/gfs/gfs.0p25.2022092400.f024.grib2
ERROR  : 

This task it to handle missing data that falls into 3 categories:

1. The input file exists but the requested field does not. In this case, MET should print a warning message and populate the cylindrical coordinate field with bad data.
2. The input file does not exist. In this case, MET should print a warning message about the missing file and populate the cylindrical coordinates with bad data for ALL fields.
3. The track ends prior to the expected duration (e.g. 0 to 126 hours every 6 hours, by default). In this case, MET should print a warning message about lead times that are not present in the track and populate the cylindrical coordinates with bad data for ALL fields.

Some details:

• Update the default TC-Diag config file to explicitly list the lead times to be processed, from 0 to 126 hours, every 6 hours.
• For missing fields (1), missing input files (2), and missing track points (3), still write missing data values to the cylindrical coordinates files.
• For (2) and (3), the temporary NetCDF files will contain ALL missing data. In this case, recommend calling the python diagnostics script WITHOUT specifying the temporary NetCDF file. That'll trigger the diagnostics script to just return all missing data values.

Time Estimate

2 or 3 days?

Sub-Issues

Consider breaking the enhancement down into sub-issues.
Hopefully none needed

Relevant Deadlines

Ideally complete this prior to Aug 1, 2023 so we can start #2550 at that point.

Funding Source

2700043

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED CYCLE ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy
• Update the TC-Diag wrapper to support updates for MET version 12.0.0 METplus#2340: Update the TC-Diag wrapper to support new config file option(s)

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.