Skip to content

Conversation

hsoh-u
Copy link
Collaborator

@hsoh-u hsoh-u commented Jun 13, 2025

Expected Differences

The main change is including upper bound for the range of indexes (0:10 ==> include 11 levels instead of 10 levels) to make the same behavior with GRIB.

  • Support two @ notations at time and vlevel

  • Other bug fixes with @Notation and range of offsets

  • Allow vertical level range for the plot_data_plane. It will give a log message which level was selected

  • Do these changes introduce new tools, command line arguments, or configuration file options? [No]

    If yes, please describe:

  • Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]

    If yes, please describe:

Pull Request Testing

  • Describe testing already performed for these changes:

  • Modified plot_data_plane unitest (offset 0 to time string).

  • Modified plot_data_plane_NCCF_latlon_by_value with two @ notations: name = "edr"; level = "(@20130730_00,@20000,*,*)";'

./grid_stat wrfprs_ruc13_03.tm00_G212 wrfprs_ruc13_03.tm00_G212 GridStatConfig.test -outdir out -v 10

note: GridStatConfig.test was from /d1/projects/METplus/discussions/2970/GridStatConfig.
Other data and configuration files were from /d1/projects/METplus/discussions/2970

netcdf era5_mnth.mrg1_avg.1440x0721.Y1985-2014.T00z {
dimensions:
        time = UNLIMITED ; // (12 currently)
        lev = 37 ;
        lat = 721 ;
        lon = 1440 ;

field = [ { name = "u"; level = "(0:10,0,*,*)"; } ]; ==> to test inclusive range

DEBUG 7: VarInfoNcCF::set_magic() ->  start: 0, end: 10
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 0
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 1
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 2
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 3
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 4
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 5
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 6
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 7
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 8
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 9
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 10
DEBUG 7: MetNcCFDataFile::collect_time_offsets(VarInfo &) -> Found 11 times between 19850115_000000 and 19851211_000000

field = [ { name = "u"; level = "(0:21,0,*,*)"; } ]; ==> upper bound of time is beyond time dimension

DEBUG 7: VarInfoNcCF::set_magic() ->  start: 0, end: 21
DEBUG 7: MetNcCFDataFile::collect_time_offsets(VarInfo &) -> Ignored index above dimension size (12)
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 0
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 1
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 2
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 3
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 4
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 5
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 6
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 7
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 8
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 9
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 10
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 11
DEBUG 7: MetNcCFDataFile::collect_time_offsets(VarInfo &) -> Found 12 times between 19850115_000000 and 19851211_000000

field = [ { name = "u"; level = "(0,0-50,*,*)"; } ];

DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 7: MetNcCFDataFile::collect_z_offsets(VarInfo &) -> Ignored index above dimension size (37)
...
DEBUG 9: MetNcCFDataFile::collect_z_offsets(VarInfo &) ->  added index 37
DEBUG 7: MetNcCFDataFile::collect_z_offsets(VarInfo &) -> Found 38 vlevels between 1 and 1000
D

field = [ { name = "u"; level = "(0:5,@500,*,*)"; } ];

DEBUG 7: MetNcCFDataFile::convert_z_to_offset() -> Found "lev" dimension value of "500" at dimension index 21.
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 0
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 1
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 2
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 3
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 4
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  added index 5
DEBUG 7: MetNcCFDataFile::collect_time_offsets(VarInfo &) -> Found 6 times between 19850115_000000 and 19851211_000000

field = [ { name = "u"; level = "(@19850714:19850813,@500,*,*)"; } ];

DEBUG 7: MetNcCFDataFile::convert_z_to_offset() -> Found "lev" dimension value of "500" at dimension index 21.
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  found the first time 19850714_000000 level lower: [19850714_000000]
DEBUG 9: MetNcCFDataFile::collect_time_offsets(VarInfo &) ->  found the time 19850813_000000
DEBUG 7: MetNcCFDataFile::collect_time_offsets(VarInfo &) -> Found 2 times between 19850115_000000 and 19851211_000000
D

field = [ { name = "u"; level = "(@19850115,@400-800,*,*)"; } ];
field = [ { name = "u"; level = "(@19850115,@400-@800,*,*)"; } ];

DEBUG 7: MetNcCFDataFile::convert_time_to_offset() -> Found 19850115_000000 at index 0 from time value
DEBUG 9: MetNcCFDataFile::collect_z_offsets(VarInfo &) ->  found the first z 400 lower: [400]
DEBUG 9: MetNcCFDataFile::collect_z_offsets(VarInfo &) ->  found the z 450
DEBUG 9: MetNcCFDataFile::collect_z_offsets(VarInfo &) ->  found the z 500
DEBUG 9: MetNcCFDataFile::collect_z_offsets(VarInfo &) ->  found the z 550
DEBUG 9: MetNcCFDataFile::collect_z_offsets(VarInfo &) ->  found the z 600
DEBUG 9: MetNcCFDataFile::collect_z_offsets(VarInfo &) ->  found the z 650
DEBUG 9: MetNcCFDataFile::collect_z_offsets(VarInfo &) ->  found the z 700
DEBUG 9: MetNcCFDataFile::collect_z_offsets(VarInfo &) ->  found the z 750
DEBUG 9: MetNcCFDataFile::collect_z_offsets(VarInfo &) ->  found the z 775
DEBUG 9: MetNcCFDataFile::collect_z_offsets(VarInfo &) ->  found the z 800
DEBUG 7: MetNcCFDataFile::collect_z_offsets(VarInfo &) -> Found 10 vlevels between 1 and 1000

Range of vertical levels for plot_data_plane:

bin/plot_data_plane       /d1/projects/MET/MET_test_data/unit_test/model_data/nccf/gtg/lc/gtg_obs_forecast.20130404.i12.f06.nc       /d1/personal/hsoh/MET/test_output/bugfix_3164_slice_range/plot_data_plane/gtg_obs_forecast.20130404.i12.f06.NCCF_lc_25.ps       'name = "edr"; level = "(@20130404_18,0-1,*,*)";'       -title "NCCF Lambert Conformal 1000ft"       -v 9

DEBUG 1: Start plot_data_plane V12.1.0 by hsoh(9895) at 2025-06-13 18:44:52Z with command: bin/plot_data_plane /d1/projects/MET/MET_test_data/unit_test/model_data/nccf/gtg/lc/gtg_obs_forecast.20130404.i12.f06.nc /d1/personal/hsoh/MET/test_output/bugfix_3164_slice_range/plot_data_plane/gtg_obs_forecast.20130404.i12.f06.NCCF_lc_25.ps name = "edr"; level = "(@20130404_18,0-1,*,*)"; -title NCCF Lambert Conformal 1000ft -v 9

==>
DEBUG 1: MetNcCFDataFile::find_z_offset() -> the lowerest level [0] was selected between 0 and 1
  • Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

  • Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [No]

  • Do these changes include sufficient testing updates? [No]

  • Will this PR result in changes to the MET test suite? [No]

    If yes, describe the new output and/or changes to the existing output:

  • Will this PR result in changes to existing METplus Use Cases? [Yes or No]

    If yes, create a new Update Truth METplus issue to describe them.

  • Do these changes introduce new SonarQube findings? [No]

    If yes, please describe:

  • Please complete this pull request review by [Fill in date].

Pull Request Checklist

See the METplus Workflow for details.

  • Review the source issue metadata (required labels, projects, and milestone).
  • Complete the PR definition above.
  • Ensure the PR title matches the feature or bugfix branch name.
  • Define the PR metadata, as permissions allow.
    Select: Reviewer(s) and Development issue
    Select: Milestone as the version that will include these changes
    Select: METplus-X.Y Support project for bugfix releases or MET-X.Y Development project for the next coordinated release
  • After submitting the PR, select the ⚙️ icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
  • After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
  • Close the linked issue and delete your feature or bugfix branch from GitHub.

Howard Soh and others added 13 commits May 28, 2025 16:28
* #3164 Fix the SonarQUbe findings

* #3164 Corrected const declarations

* #3164 Corrected const declarations

* #3164 Fixes for SonarQube findings

* #3164 Fixes for SonarQube findings

* #3164 Fixes for SonarQube findings

* #3164 Fixes for SonarQube findings: const

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>
* #3164 Fix the SonarQUbe findings

* #3164 Corrected const declarations

* #3164 Corrected const declarations

* #3164 Fixes for SonarQube findings

* #3164 Fixes for SonarQube findings

* #3164 Fixes for SonarQube findings

* #3164 Fixes for SonarQube findings: const

* Renamed error_code_bad_offset to error_code_out_of_index

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>
@github-project-automation github-project-automation bot moved this to 🩺 Needs Triage in METplus-6.2 Development Jun 16, 2025
@JohnHalleyGotway JohnHalleyGotway moved this from 🩺 Needs Triage to 🔎 In review in METplus-6.2 Development Jun 16, 2025
@JohnHalleyGotway JohnHalleyGotway modified the milestone: MET-12.2.0 Jun 16, 2025
@hsoh-u hsoh-u requested a review from JohnHalleyGotway June 16, 2025 14:53
@JohnHalleyGotway JohnHalleyGotway requested review from JohnHalleyGotway and removed request for JohnHalleyGotway June 16, 2025 14:57
@hsoh-u hsoh-u mentioned this pull request Jun 16, 2025
17 tasks
@JohnHalleyGotway JohnHalleyGotway changed the title Bugfix 3164 slice range Bugfix #3164 develop slice range Jun 18, 2025
Copy link
Collaborator

@JohnHalleyGotway JohnHalleyGotway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that I did propose a few minor edits for consistent formatting.
I tested in seneca /d1/projects/MET/MET_pull_requests/met-12.2.0/rc1/MET-bugfix_3164_slice_range. With plot_data_plane and I found slightly inconsistent behavior for handling bad inputs:

cd /d1/projects/MET/MET_pull_requests/met-12.2.0/rc1/MET-bugfix_3164_slice_range

# Bad time request results in a library ERROR
bin/plot_data_plane  $MET_TEST_INPUT/model_data/nccf/gtg/latlon/gtg_obs_forecast.20130730.i00.f00.nc plot.ps 'name="edr"; level="(@20130731,@380,*,*)";'
ERROR  : MetNcCFDataFile::find_time_offset() -> the requested time 20130731_000000 for "edr" variable does not exist (20130730_000000 and 20130730_000000).

# Bad level request results in a library WARNING
bin/plot_data_plane  $MET_TEST_INPUT/model_data/nccf/gtg/latlon/gtg_obs_forecast.20130730.i00.f00.nc plot.ps 'name="edr"; level="(@20130730,@380,*,*)";'
WARNING: MetNcCFDataFile::find_z_offset() -> the requested vlevel 380 for "edr" variable does not exist (20000 and 45000).

Should these both be warnings? Or both be errors?

I also note:

  1. No differences are flagged by GHA.
  2. SonarQube code smells are reduced by this PR by 32 (from 15,617 to 15,585).
  3. Existing unit tests are modified to demonstrate that replacing the time index with YYYYMMDD_HH notation OR @YYYYMMDD_HH notation produces the same result.
  4. Manually ran a few tests to confirm that it behaves as expected with good inputs.

Copy link
Collaborator

@JohnHalleyGotway JohnHalleyGotway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate

@hsoh-u
Copy link
Collaborator Author

hsoh-u commented Jun 23, 2025

Note that I did propose a few minor edits for consistent formatting. I tested in seneca /d1/projects/MET/MET_pull_requests/met-12.2.0/rc1/MET-bugfix_3164_slice_range. With plot_data_plane and I found slightly inconsistent behavior for handling bad inputs:

cd /d1/projects/MET/MET_pull_requests/met-12.2.0/rc1/MET-bugfix_3164_slice_range

# Bad time request results in a library ERROR
bin/plot_data_plane  $MET_TEST_INPUT/model_data/nccf/gtg/latlon/gtg_obs_forecast.20130730.i00.f00.nc plot.ps 'name="edr"; level="(@20130731,@380,*,*)";'
ERROR  : MetNcCFDataFile::find_time_offset() -> the requested time 20130731_000000 for "edr" variable does not exist (20130730_000000 and 20130730_000000).

# Bad level request results in a library WARNING
bin/plot_data_plane  $MET_TEST_INPUT/model_data/nccf/gtg/latlon/gtg_obs_forecast.20130730.i00.f00.nc plot.ps 'name="edr"; level="(@20130730,@380,*,*)";'
WARNING: MetNcCFDataFile::find_z_offset() -> the requested vlevel 380 for "edr" variable does not exist (20000 and 45000).

Should these both be warnings? Or both be errors?

I also note:

  1. No differences are flagged by GHA.
  2. SonarQube code smells are reduced by this PR by 32 (from 15,617 to 15,585).
  3. Existing unit tests are modified to demonstrate that replacing the time index with YYYYMMDD_HH notation OR @YYYYMMDD_HH notation produces the same result.
  4. Manually ran a few tests to confirm that it behaves as expected with good inputs.

It's an error and will be stopped.

Note that I did propose a few minor edits for consistent formatting. I tested in seneca /d1/projects/MET/MET_pull_requests/met-12.2.0/rc1/MET-bugfix_3164_slice_range. With plot_data_plane and I found slightly inconsistent behavior for handling bad inputs:

cd /d1/projects/MET/MET_pull_requests/met-12.2.0/rc1/MET-bugfix_3164_slice_range

# Bad time request results in a library ERROR
bin/plot_data_plane  $MET_TEST_INPUT/model_data/nccf/gtg/latlon/gtg_obs_forecast.20130730.i00.f00.nc plot.ps 'name="edr"; level="(@20130731,@380,*,*)";'
ERROR  : MetNcCFDataFile::find_time_offset() -> the requested time 20130731_000000 for "edr" variable does not exist (20130730_000000 and 20130730_000000).

# Bad level request results in a library WARNING
bin/plot_data_plane  $MET_TEST_INPUT/model_data/nccf/gtg/latlon/gtg_obs_forecast.20130730.i00.f00.nc plot.ps 'name="edr"; level="(@20130730,@380,*,*)";'
WARNING: MetNcCFDataFile::find_z_offset() -> the requested vlevel 380 for "edr" variable does not exist (20000 and 45000).

Should these both be warnings? Or both be errors?

I also note:

  1. No differences are flagged by GHA.
  2. SonarQube code smells are reduced by this PR by 32 (from 15,617 to 15,585).
  3. Existing unit tests are modified to demonstrate that replacing the time index with YYYYMMDD_HH notation OR @YYYYMMDD_HH notation produces the same result.
  4. Manually ran a few tests to confirm that it behaves as expected with good inputs.

It should be an error and be stopped in order to avoid to read a NetCDF variable with bad offset. It's fixed

Copy link
Collaborator

@JohnHalleyGotway JohnHalleyGotway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approve of these changes.

Thanks for updating the logic to consistently error our for bad vertical level or timestring values. I confirmed that both cases error out as expected:

ERROR  : MetNcCFDataFile::find_time_offset() -> the requested time 20130731_000000 for "edr" variable does not exist (20130730_000000 and 20130730_000000).
...
ERROR  : MetNcCFDataFile::find_z_offset() -> the requested vlevel 380 for "edr" variable does not exist (20000 and 45000).

@hsoh-u hsoh-u merged commit 811c494 into develop Jun 24, 2025
39 of 40 checks passed
@github-project-automation github-project-automation bot moved this from 🔎 In review to 🏁 Done in METplus-6.2 Development Jun 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 🏁 Done
Development

Successfully merging this pull request may close these issues.

Bugfix: Correct NetCDF dimension range checking to use inclusive inequalities
2 participants