Skip to content

Conversation

JohnHalleyGotway
Copy link
Collaborator

@JohnHalleyGotway JohnHalleyGotway commented Aug 12, 2025

Expected Differences

The problem here is that calls to Python 3.10's version of Py_InitializeFromConfig clobber the Python path. Calls to the Python 3.12 version DO NOT have this problem. The fix is to consolidate calls to Py_InitializeFromConfig in one location and store/restore the Python path before/after calling Py_InitializeFromConfig.

These changes include:

  1. Define new GlobalPython::set_args() function in global_python.h to handle the stashing and restoring of the Python system path in only one spot.
  2. Replace existing calls to PyConfig_SetArgv with a calls to GlobalPython::set_args() in the following locations:
  • python_dataplane.cc when processing gridded data
  • python_pointdata.cc when processing point observations
  • python_tc_diag.cc when computing tropical cyclone diagnostics

Note that I'll cherry-pick these changes into a PR for develop as well, but will look for additional opportunities to cleanup and standardize the Python embedding function names and logging.

  • Do these changes introduce new tools, command line arguments, or configuration file options? [No]

    If yes, please describe:

  • Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]

    If yes, please describe:

Pull Request Testing

  • Describe testing already performed for these changes:

    Compiled using Python 3.10 on seneca and ran all of the unit tests there. Note that GHA builds/tests using Python 3.12 (from METbaseimage verison 3.4.2). But the Python 3.10 testing is done on seneca.

  • Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

  1. Confirm that all the Python 3.12 unit tests pass through GHA.
  2. Inspect Python 3.10 unit test results on seneca to confirm that all tests run and no differences are present.
# On seneca
egrep -i "warning|error" /d1/projects/MET/MET_pull_requests/met-12.1.1/MET-bugfix_3219_main_v12.1_python/internal/test_unit/unit_test_py310.log
egrep -i "warning|error" /d1/projects/MET/MET_pull_requests/met-12.1.1/MET-bugfix_3219_main_v12.1_python/internal/test_unit/comp_dir_py310.log

Here's the 3 diffs flagged:

COMPARING netcdf/regrid_data_plane_months_units_day2.nc
ERROR: NetCDF file size difference exceeds 1 %
COMPARING netcdf/regrid_data_plane_months_units.nc
ERROR: NetCDF file size difference exceeds 1 %
COMPARING netcdf/regrid_data_plane_years_units.nc
ERROR: NetCDF file size difference exceeds 1 %

The data is the same, but the NetCDF file sizes differ slightly, which is not a real concern.

  • Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]
    None needed.

  • Do these changes include sufficient testing updates? [Yes]
    No new unit tests are needed. To actually test these changes we could create a new METbaseimage 3.4.2_py310 image and compile/test MET with that. But that's beyond the scope of a funding.

  • Will this PR result in changes to the MET test suite? [No]

    If yes, describe the new output and/or changes to the existing output:

  • Will this PR result in changes to existing METplus Use Cases? [No]

    If yes, create a new Update Truth METplus issue to describe them.

  • Do these changes introduce new SonarQube findings? [Yes or No]

    If yes, please describe:

  • Please complete this pull request review by [Fill in date].

Pull Request Checklist

See the METplus Workflow for details.

  • Review the source issue metadata (required labels, projects, and milestone).
  • Complete the PR definition above.
  • Ensure the PR title matches the feature or bugfix branch name.
  • Define the PR metadata, as permissions allow.
    Select: Reviewer(s) and Development issue
    Select: Milestone as the version that will include these changes
    Select: METplus-X.Y Support project for bugfix releases or MET-X.Y Development project for the next coordinated release
  • After submitting the PR, select the ⚙️ icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
  • After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
  • Close the linked issue and delete your feature or bugfix branch from GitHub.

…ath prior to calling Py_InitializeFromConfig() and then restore the Python path afterwards. For Python 3.10, the call to Py_InitializeFromConfig() resets the Python path while in Python 3.12 it does not.
@JohnHalleyGotway JohnHalleyGotway marked this pull request as ready for review August 12, 2025 22:43
@JohnHalleyGotway JohnHalleyGotway mentioned this pull request Aug 13, 2025
17 tasks
@JohnHalleyGotway JohnHalleyGotway removed the request for review from georgemccabe August 13, 2025 17:49
Copy link
Collaborator

@hsoh-u hsoh-u left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Built and tested at seneca with python3.10 environment.
The unittest unit_python.xml ran successfully.

@JohnHalleyGotway JohnHalleyGotway merged commit 23b5024 into main_v12.1 Aug 14, 2025
64 of 65 checks passed
@github-project-automation github-project-automation bot moved this from 🔎 In review to 🏁 Done in Coordinated METplus-6.1 Support Aug 14, 2025
JohnHalleyGotway added a commit that referenced this pull request Aug 28, 2025
* Bugfix #3165 main v12.1 intel compilation (#3167)

* Per #3165, correct two poorly formed OpenMP pragma statements.

* Per #3165, add missing array include to ensemble_stat, point_stat, and plot_point_obs that's appearently needed when compiling with the Intel OneAPI compilers and with the --disable-opemp configuration option.

* Per #3165, move 'using namespace std' after the includes in the all the test code and enum_parser.yy to avoid Intel OneAPI compilation warnings about it being implicitly defined.

* Per #3165, only compile the kdtree test programs when ENABLE_UGRID is true.

* Per #3165, only compile the kdtree test programs when ENABLE_UGRID is true.

* Per #3165, move 'using namespace std' after the includes in the all the test code.

* Per #3165, add DISABLE_OPENMP option to compilation script

* Per #3165, updating beta2 reference to rc1

* Per #3165, fix typo in internal/test_util/libcode/vx_grid/Makefile.am

* Per #3165, removing typo in acorn config file

* Per #3165, Test whether disabling custom reductions from the vx_series_data library enables MET to compile on WCOSS2 with Intel classic compilers.

---------

Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>

* Hotfix to main_v12.1 to update the rc1 release note contents and date.

* Updating versions for atlas and eckit

* per #3175, replace bits/stdc++.h include that is GCC-specific with regex which works with other compilers (#3177)

* Per #3179, update the logic to not search the parent when parsing file_type from climatology dictionaries. (#3180)

* Bugfix #3179 main_v12.1 climo_file_type TAKE 2 (#3182)

* Per #3179, update the logic to not search the parent when parsing file_type from climatology dictionaries.

* Per #3179, modify the climo_name file_type logic to parse it from the top-level climo dictionary rather than parsing it separately for each field array entry.

---------

Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>

* Update install_met_env.wcoss2

Removing "-rc1"

* Bugfix #3164 main_v12.1 slice range (#3184)

* #3164 The range selection is inclusive like GRIB (the upper offset was not included before)

* #3164 CHnaged warning to error for bad offset for Z

* #3164 Changed index offsets to value notations

* #3164 Support @level1-@lavel2

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* Per dtcenter/METbaseimage#32, hotfix to the MET main_v12.1 branch to make the docker build enviornment more portable.

* Adding modulefile for new machine ursa

* Adding config file for new machine ursa

* Feature dtcenter/METbaseimage#32 main_v12.1 release docker (#3195)

* Per dtcenter/METbaseimage#32, port over changes from main_v12.0 to main_v12.1, but update the schedule to rebuild images weekly rather than daily.

* Better comment

* Feature #3198 main_v12.1 scan_for_CVEs (#3201)

* Per #3198, port over CVE scanning logic from METbaseimage to MET.

* Per #3198, use SOURCE_BRANCH instead of GITHUB_NAME for the scan_docker_image.sh script.

* Fix typo

* Update CVE log message

* Per #3198, update the versions listed in the release-docker-images.yml workflow for the develop branch.

* Bugfix #3206 main_v12.1 AW_MEAN (#3207)

* Unrelated to #3206, increase the METbaseimage version number from v3.4 to v3.4.1 so that the resulting Docker images contain 0 critical vulnerabilities.

* Per #3206, update met_regrid_area_weighted() function to fix bug when OMP_NUM_THREADS is set very large. Need to use a reduction for vectors of doubles to prevent two threads from updating the same to_data value at the same time.

* Per #3206, add comment.

* Bugfix #3210 main_v12.1 WDIR (#3211)

* Fix jobstring to write -out_wind_logic correctly instead of reporting the value for cnt_logic instead.

* Restore logic in aggr_mpr_wind_lines() removed for #2395 when STAT-Analysis aggregates MPR lines to compute WDIR output. In main_v11.0, that job automatically removes 0 vectors when computing ROW_MEAN_WDIR output.

* Per #3210, update the revision history

* Per #3210, update the WDIR documentation

* Per #3210, add one more WDIR job to STAT-Analysis using all inputs, not just the non-zero MPR vectors.

* Unrelated to #3210, but noted by @CPKalb, note that MET can read raw WRF output.

* Per #3210, solve one additional bug in the MPR -> VCNT aggregation. Since the local v_info object was declared outside the loop, when d_diff is bad data, the values from the previous iteration were being using. This cause the TOTAL_DIR value to be reported as the TOTAL value and not properly skip over the zero length vectors as it should... and as Point-Stat does.

---------

Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>

* Feature #3191 v12.1.0 (#3217)

* Per #3191, update GitHub actions logic prior to the MET-12.1.0 release to use consistent indentation, switch to using , update scan_docker_image.sh to fail on critical CVEs, update the cve scanning function to print a summary, and update the versions of MET to be rebuilt on schedule.

* Per #3191, update the METbaseimage version from 'v3.4.1' to '3.4-latest' to enable security updates in METbaseimage to be incorparated into future rebuilds of the MET image.

* Per #3191, consolidate MET-12.1.0 release notes.

* Per #3191, more updates to docs for the 12.1.0 release.

* Per #3191, add back in the MET-12.0.Z release notes since we only remove old release notes when we increase the major version number.

* Per #3191, refine GHA job logic and log formatting.

* Per #3191, switch to using ENV a=b instead of deprecated ENV a b

* In installation.rst change python 3.10 to 3.12

* change release date

---------

Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>

* Bugfix #3219 main_v12.1 python (#3220)

* Per #3219, add GlobalPython:set_args() function to store the Python path prior to calling Py_InitializeFromConfig() and then restore the Python path afterwards. For Python 3.10, the call to Py_InitializeFromConfig() resets the Python path while in Python 3.12 it does not.

* Per #3219, update logic to eliminate duplicated log messages.

* Hotfix to `release-docker-images.yml` to disable fail-fast.

* Per #3228, updates for the METv12.1.1 bugfix release, including release notes, doc version, code version, and the release docker images workflow. (#3229)

Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>

---------

Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>
Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
Co-authored-by: Howard Soh <hsoh@ucar.edu>
Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>
Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

Bugfix: Python embedding does not work for MET version 12.1.0 with Python version 3.10
2 participants