Feature dtcenter/METbaseimage#30 changes to support py3.12 #3091

georgemccabe · 2025-02-24T17:35:51Z

These are changes that were required to get the MET unit tests to pass when I switched to using Python 3.12.0. I reverted the changes that use Python 3.12.0 to ensure that these changes don't break the current environment that uses Python 3.10.

Note: I defined the NumpyTypeEncoder in 2 places because I wasn't sure how the imports work and which directories are available to import in each file. The copy defined in scripts/python/met/logger.py is used in 2 places, but it is also needed in scripts/python/pyembed/python_embedding.py. I wasn't sure if logger.py was the appropriate file to include this, but it looks like that file contains other shared logic besides logging, so I added it there.

These changes should result in no differences in the MET unit tests and METplus use cases.
The changes appears to cause rounding differences in the NetCDF lat/lon fields when using Python 3.12.0. See this GHA run to view those differences: https://github.com/dtcenter/METplus/actions/runs/13419641478

Example:

COMPARING met_tool_wrapper/PCPCombine_python_embedding/met_tool_wrapper/PCPCombine/PCPCombine_combine_py_embed/IMERG.20180102_1300_A06h
file_A: /data/truth/met_tool_wrapper/PCPCombine_python_embedding/met_tool_wrapper/PCPCombine/PCPCombine_combine_py_embed/IMERG.20180102_1300_A06h
file_B: /data/output/met_tool_wrapper/PCPCombine_python_embedding/met_tool_wrapper/PCPCombine/PCPCombine_combine_py_embed/IMERG.20180102_1300_A06h

Using default rounding precision 6
Comparing NetCDF
ERROR: Field (lat) values differ
Min diff: -7.62939453125e-06, Max diff: 0.0
ERROR: Field (lon) values differ
Min diff: -1.52587890625e-05, Max diff: 0.0

Once this PR and the METbaseimage PR are merged, we can apply the updates from branch feature_baseimage_30_py3.12.0 to switch to actually using Python 3.12.0.

Expected Differences

Do these changes introduce new tools, command line arguments, or configuration file options? [No]

If yes, please describe:
Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]

If yes, please describe:

Pull Request Testing

Describe testing already performed for these changes:

Tested that all MET unit tests and METplus use cases succeed using these changes and Python 3.12.0. Reverted the Python version back to 3.10 and tested again to ensure that these changes do not break anything in the current version.

Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

Review code changes, advise on better locations for the shared python logic.

Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]
Do these changes include sufficient testing updates? [Yes]
Will this PR result in changes to the MET test suite? [No]

If yes, describe the new output and/or changes to the existing output:
Will this PR result in changes to existing METplus Use Cases? [No]

If yes, create a new Update Truth METplus issue to describe them.
Do these changes introduce new SonarQube findings? [No]

If yes, please describe:
Please complete this pull request review by 2/27/2025.

Pull Request Checklist

See the METplus Workflow for details.

Review the source issue metadata (required labels, projects, and milestone).
Complete the PR definition above.
Ensure the PR title matches the feature or bugfix branch name.
Define the PR metadata, as permissions allow.
Select: Reviewer(s) and Development issue
Select: Milestone as the version that will include these changes
Select: Coordinated METplus-X.Y Support project for bugfix releases or MET-X.Y.Z Development project for official releases
After submitting the PR, select the ⚙️ icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
Close the linked issue and delete your feature or bugfix branch from GitHub.

… python 3.10 to 3.12.0

…json to prevent error handling float32 values

…s://gist.github.com/jonathanlurie/1b8d12f938b400e54c1ed8de21269b65

…2 calls to json.dump

…ed in fire weather use case moving to numpy 2.X

…ogic are backwards compatible

georgemccabe · 2025-03-07T22:09:06Z

@hsoh-u , @DanielAdriaansen , just following up on this open PR

DanielAdriaansen · 2025-03-10T19:59:33Z

@georgemccabe I reviewed the changes here, but I don't quite understand the nuances fully. @hsoh-u would be best to comment on the actual changes to the Python layer in MET that are included here (as you note by suggesting for reviewers "Review code changes, advise on better locations for the shared python logic.")

I don't feel concerned about the differences in lat/lon that these changes introduced.

hsoh-u

I approve the changes.
The python packages "met" were designed as the base package for the python embedding. The dependency from the met package to the pyembed package is expected, but not for the opposite direction. So read_tmp_data_plane.py imports met.dataplane and read_tmp_point_nc.py imports met.point. So the same class at the python_embedding.py can import NumpyTypeEncoder from met.logger.

…port it from the other location

georgemccabe · 2025-03-12T17:50:40Z

Thanks, @hsoh-u. I just pushed a commit to remove the duplicate class and instead import it from met.logger. If all of the tests pass, I will merge this PR.

…stead import it from the other location" This reverts commit b489e79.

georgemccabe · 2025-03-12T18:21:18Z

@hsoh-u , importing met.logger from python_embedding.py resulted in an import error: https://github.com/dtcenter/MET/actions/runs/13818134931/job/38657704489?pr=3091#step:5:150
I will revert this change back

hsoh-u · 2025-03-12T18:49:37Z

@hsoh-u , importing met.logger from python_embedding.py resulted in an import error: https://github.com/dtcenter/MET/actions/runs/13818134931/job/38657704489?pr=3091#step:5:150 I will revert this change back

Sorry for the trouble. Let's keep it simple with revert.
FYI: The location of import matters, and it has to be imported at the bottom of add_python_path() (before return).

georgemccabe added 16 commits February 4, 2025 13:49

per dtcenter/METbaseimage#30, update variables to reflect change from…

d24a84d

… python 3.10 to 3.12.0

set met base tag to test

77879df

remove warning that $ is invalid

79b093d

print exception error so it is easier to debug failures in script

82b4de7

change np.NaN to np.nan because the former was deprecated in numpy 2.0

834b94c

convert python embedding attributes to a string before converting to …

07ac764

…json to prevent error handling float32 values

define json encoder to convert numpy types to python -- see gist http…

9545467

…s://gist.github.com/jonathanlurie/1b8d12f938b400e54c1ed8de21269b65

move numpy json encoder to common area where it can be referenced by …

4ea7407

…2 calls to json.dump

add encoder to python embedding script (to test, should move later)

f82db67

update /nrit/ral python to use 3.12

a726f02

Merge branch 'develop' into feature_baseimage_30_py3.12.0

83acd35

convert numpy types to python in the attributes to fix issue introduc…

f5c488e

…ed in fire weather use case moving to numpy 2.X

revert back to python 3.10 to test that changes to python embedding l…

9ca6650

…ogic are backwards compatible

revert back to python 3.10 to test

c1799dd

Merge branch 'develop' into feature_baseline_30_test_py310

938dad0

change python settings back to 3.10 for testing on seneca

dbca25e

georgemccabe added this to the MET-12.1.0 milestone Feb 24, 2025

georgemccabe requested review from hsoh-u and DanielAdriaansen February 24, 2025 17:35

hsoh-u previously approved these changes Mar 11, 2025

View reviewed changes

georgemccabe added 2 commits March 12, 2025 11:47

Merge branch 'develop' into feature_baseline_30_test_py310

33c015e

per PR feedback from @hsoh-u, removing duplicate class and instead im…

b489e79

…port it from the other location

georgemccabe dismissed hsoh-u’s stale review via b489e79 March 12, 2025 17:49

Revert "per PR feedback from @hsoh-u, removing duplicate class and in…

b90400e

…stead import it from the other location" This reverts commit b489e79.

fix SonarQube complaints

f46a257

georgemccabe merged commit 3c7e08c into develop Mar 12, 2025
39 of 40 checks passed

github-project-automation bot moved this from 🔎 In review to 🏁 Done in METplus-6.1 Development Mar 12, 2025

georgemccabe deleted the feature_baseline_30_test_py310 branch March 12, 2025 20:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature dtcenter/METbaseimage#30 changes to support py3.12 #3091

Feature dtcenter/METbaseimage#30 changes to support py3.12 #3091

Uh oh!

georgemccabe commented Feb 24, 2025

Uh oh!

georgemccabe commented Mar 7, 2025

Uh oh!

DanielAdriaansen commented Mar 10, 2025 •

edited

Loading

Uh oh!

hsoh-u left a comment

Uh oh!

georgemccabe commented Mar 12, 2025

Uh oh!

georgemccabe commented Mar 12, 2025

Uh oh!

hsoh-u commented Mar 12, 2025

Uh oh!

Uh oh!

Uh oh!

Feature dtcenter/METbaseimage#30 changes to support py3.12 #3091

Feature dtcenter/METbaseimage#30 changes to support py3.12 #3091

Uh oh!

Conversation

georgemccabe commented Feb 24, 2025

Expected Differences

Pull Request Testing

Pull Request Checklist

Uh oh!

georgemccabe commented Mar 7, 2025

Uh oh!

DanielAdriaansen commented Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hsoh-u left a comment

Choose a reason for hiding this comment

Uh oh!

georgemccabe commented Mar 12, 2025

Uh oh!

georgemccabe commented Mar 12, 2025

Uh oh!

hsoh-u commented Mar 12, 2025

Uh oh!

Uh oh!

Uh oh!

DanielAdriaansen commented Mar 10, 2025 •

edited

Loading