Update develop-ref after #3036 #3046
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> (cherry picked from commit 44335f3) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu>
* run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report
* Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
* Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
* Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe
* per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe822. * per #2567, fix bug that caused file window logic to fail
* Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg
…ily disabled land_surface:0 until we can resolve the differences
* Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting
…to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update
…tead merge changes from -ref into the update branch
… Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- #2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward
* prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM
* Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
* Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page.
* per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated
* clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation
* added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
* Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu>
* per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting
* Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
* added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu>
* Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use case config gile to remove unused items. * Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min. * Removes unused functions from forecast Python embedding scripts. * Switches to F strings and wraps some print statements in DEBUG flag. * Switches to F strings and adds script name to print statements. * Adds space around print statements to aid log file inspection. * Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file. * Adds controlling DEBUG level for observation Python embedding scripts. * Updates documentation URL in config file. * Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA. * Updates for sample data and running the use case in GHA. * Typo in the JSON file. * Changes the MET_PYTHON_DIR to the installed location rather than the source code location. * Adds user script to create graphic from PointStat MPR output. * Adds use case gallery image and updates observation dataset info in documentation. * Fixes path to config file include in documentation. * Adds new user script for plotting to the documentation file. * Updates list of expected output files to include UserScript outpout and PB2NC output. * Changes to eliminate code smells. * Finalizes forecast dataset description and scientific objective. * Adds info on additional Python packages required for Python embedding and UserScripts. * Removes extra #. * Removes uneccesary imports in user script. * Adds more clarity that METcalcpy is a required Python package for Python embedding. * Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts. * Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3. * Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts).
… periodically to fix any security vulnerabilities in upstream images
Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu>
* Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extension * add more information for use case doc * turn off use case for PR * refactor map functions script to reduce cognitive complexity for SonarQube * update scientific objective with input from @jaredalee * update use case to reference environment variables in reformatted config file instead of using relative paths and changing directories since changes needed to support env vars in the reformatted are available in develop now * test that changes to use case did not break anything
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#3036 adds the tc_and_extra_tc:6 use case for WRF Hurricane Matthew
Created by @georgemccabe