Describe the Problem

The issue arose from METplus Discussion #1189 . The madis2nc tool rejects certain mesonet data from MADIS due to a change in variables/format to the NetCDF files. Sometime in 2016, a change was made to the mesonet output files where they no longer contain variables such as precip3hr, precip6hr, etc. The MADIS support team and confirmed this was an intentional update and said that they do add/change variables to the raw data from time to time, which complicates the current methods for identifying MADIS data type.

The change in data files can be demonstrated in the following files. On sencea in /d1/projects/METplus/discussions/1189, I ran the following

wget https://madis-data.ncep.noaa.gov/madisPublic1/data/archive/2016/01/01/LDAD/mesonet/netCDF/20160101_0000.gz
gunzip 20160101_0000.gz
../../../MET/MET_releases/met-10.1.0-beta2/bin/madis2nc 20160101_0000 20160101_0000.nc -type mesonet

and confirmed this produced a successful run.

wget https://madis-data.ncep.noaa.gov/madisPublic1/data/archive/2017/01/01/LDAD/mesonet/netCDF/20170101_0000.gz
gunzip 20170101_0000.gz
../../../MET/MET_releases/met-10.1.0-beta2/bin/madis2nc 20170101_0000 20170101_0000.nc -type mesonet

and confirmed that this produced the following error and warnings:

DEBUG 1: Reading MADIS File:	20170101_0000
ERROR  : 
ERROR  : process_madis_mesonet() Please check if the input is a MESONET.
ERROR  : 
WARNING:     missing variable: precip3hr
WARNING:     missing variable: precip6hr
WARNING:     missing variable: precip12hr
WARNING:     missing variable: precip10min

We want to fix main_v10.0 for MET version 10.0.1 and develop for MET version 10.1.0.

Expected Behavior

MADIS2NC should be able to read various mesonet data formats from MADIS.

Environment

Describe your runtime environment:
1. Machine: (e.g. HPC name, Linux Workstation, Mac Laptop) Seneca
2. OS: (e.g. RedHat Linux, MacOS) Linux
3. Software version number(s)

To Reproduce

Describe the steps to reproduce the behavior: (see above)
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
Post relevant sample data following these instructions:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-help-desk#ftp

Relevant Deadlines

MET-10.1.0-beta4 (11/15/21)

Funding Source

2702691

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Organization level Project for support of the current coordinated release
• Select Repository level Project for development toward the next official release or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

Bugfix Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of main_<Version>.
  Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
• Fix the bug and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into main_<Version>.
  Pull request: bugfix <Issue Number> main_<Version> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Organization level software support Project for the current coordinated release
  Select: Milestone as the next bugfix version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Complete the steps above to fix the bug on the develop branch.
  Branch name: bugfix_<Issue Number>_develop_<Description>
  Pull request: bugfix <Issue Number> develop <Description>
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Close this issue.