Update develop-ref after #3037 #3097
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting
…to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update
…tead merge changes from -ref into the update branch
… Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- #2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward
* prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM
* Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
* refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc
* change wording of error log to satisfy test
* set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup
* refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors
* refactor duplicate code into function to satisfy SonarQube
* removed unused variable
* use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method
* fix typo in key
* suppress warnings when files aren't found because it is expected
* formatting to be consistent in doc string
* per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs
* add documentation blocks for new functions, ci-run-all-diff
* use pytest fixture instead of local function
* add support for setting -vld_thresh argument
* per #2578, added documentation and tests for setting -input_thresh and -vld_thresh
* moved verbosity argument to end of command to more easily change it when debugging
* refactor to reduce cognitive complexity to satisfy SonarQube
* update usage statement to include METplus version number
* add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired
Update METplus-6.0.0 development schedule.
* Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification
…command to upload asset
* Per #2656, update the read_iodav2_mpr.py scripts to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Per #2656, update the ugrid_lfric_mpr.py to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns.
* Per 2679, updating user support documentation * Per #2679, fixing formatting error * Update user_support.rst * Update user_support.rst --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
* Per #2452, updating MET official release instructions * Update to indicate the "DTC" web server machine Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Updated language for tar file Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
* Refactor to prevent calling sys.exit from functions that could cause the calling script to exit unintentionally. Instead return None or False and only sys.exit a non-zero value from a script that is called explicitly. Refactored validate config logic to reduce cognitive complexity to satisfy SonarQube * per #2596, ensure the same RUN_ID is set for a given METplus run even if instances are used in the PROCESS_LIST * fixed linter complaints * remove unused imports * cleanup for SonarQube and linter * remove unused imports * import parent wrappers directly so that PyCharm can follow class hierarchy * per #2612, add support for processing groups of forecast leads * remove call to prune_empty function and ci-run-all-diff * fixed check on DataFrame to properly check for an empty one * remove prune_empty function that is no longer used * update conda environment requirements for metdataio * set label to Group<n> for LEAD_SEQ_<n> if LEAD_SEQ_<n>_LABEL is not set (instead of erroring), use forecast lead label for lead if set, log lead group label and formatted list of leads for each run * increase timeout for request from 60 to 90 * handle exception and output an error message with a suggestion to rerun the failed GHA jobs when a hiccup with DockerHub prevents the list of available Docker data volumes from being obtained. FYI @jprestop * cleanup to address SonarQube complaints, move call to exit out of main function so a script calling the function does not exit when the function errors * cleanup to appease SonarQube * per #2612, add support for creating groups of forecast leads by specifying a list of forecast leads and a time interval to create groups of leads * per #2612, add unit tests for function that gets groups of forecast leads and added tests for new groups capabilities * various refactoring to satisfy SonarQube complaints * renamed config LEAD_SEQ_DIVISIONS to LEAD_SEQ_GROUP_SIZE and LEAD_SEQ_DIVISIONS_LABEL to LEAD_SEQ_GROUP_LABEL * per #2612, add documentation for new config variables including examples * fix empty check for DataFrame * fix rst typo * minor cleanup to satisfy linter * Per #2612, if runtime freq is RUN_ONCE_PER_LEAD and groups of forecast leads are specified, then the wrapper will be run once per lead GROUP instead of once for each unique lead in the lead groups. Update unit test to reflect new behavior
…all to cartopy feature download script -- see SciTools/cartopy#2263
… periodically to fix any security vulnerabilities in upstream images
Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu>
* Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extension * add more information for use case doc * turn off use case for PR * refactor map functions script to reduce cognitive complexity for SonarQube * update scientific objective with input from @jaredalee * update use case to reference environment variables in reformatted config file instead of using relative paths and changing directories since changes needed to support env vars in the reformatted are available in develop now * test that changes to use case did not break anything
* Initial config file for this use case. * Adds ismn as an acceptable format for ASCI2NC_INPUT_FORMAT. * Latest param file changes. * Documentation file for this use case. * Updates documentation URL in config file. * Updated conf file that produces matched pairs for soil moisture from PointStat. * Updates to documentation for ISMN use case. * Adds UserScripts for plotting. * Latest conf file settings for use case. * Use case gallery image. * Adds summary using NEAREST for point_stat matched pairs. * Adds path to gallery image for use case. * Changes use case template to have the sphinx gallery line separate from the previous section to avoid having the actual sphinx_gallery_path appear in the rendered HTML documentation. Still need to fix use cases that don't have this change. * Per discussion with @georgemccabe and experience using the StatAnalysis Wrapper for the ISMN use case, moving two stat-analysis wrapper config items that were listed as required to the optional section since it worked without setting them. Presumably they may be required based on how a user is running stat-analysis or the stat-analysis wrapper, but they are not always required. * Per discussion with @georgemccabe, adds additional details for adding a new use case instructing the user to return the run setting to false in the use_case_groups.json file prior to merging the PR, which is the desired default setting for all use cases. * Minor change to use case documentation. * Initial work for adding ISMN to Verification Datasets guide. * Adds documentation for the ISMN dataset in the Verification Dataset Guide. * Adds VxDataISMN to use case documentation to link the use case to the verification datasets guide. * Adds new use case to testing suite after adding sample data. * Removes mention of RAL FTP site for sending sample data for a new use case, since that is not allowed anymore and generically refers to the GitHub Discussion describing the new process using Google Drive with a link to the Discussion. * Forgot a space.
* Update to MET 12.0.3 bugfix version. * Switch from using "secrets.DOCKER_USERNAME/secrets.DOCKER_PASSWORD" to using "dtcenter/secrets.DOCKER_TOKEN".
* Per #3054, enhance release-docker-images.yml to support CVE scanning, scheduled weekly runs for one or more versions, and update the logic for pushing X.Y-latest tags to Docker Hub. * Per #3054, update the Release Guide with instructions about updating the GHA release workflows. * Per #3054, fix workflow typo. * Per #3054, update release guide formatting. * Per #3054, add CMD_LOGFILE logic to the time_command function. * Per #3054, refine release guide instructions. * Per #3054, refine release guide instructions. * Per #3054, update the versions in the release-docker-images.yml workflow for version 6.1.0. * Per #3054, update to plotly 6.1.1 and kaleido 1.0.0 versions. * Per #3054, revert changes to python package versions since those do not actually impact the GHA testing environment. * Per #3054, updating plotly version to 6.1.1 and python-kaledio version to 1.0.0 in the documentation and internal scripts. Hoping that will enable the 4 failing METplus use cases to succeed.
…ETplotpy use cases
…e MET/METviewer/METexpress store downloads in a separate website location than the other components. (#3074)
…3075) * update instructions to create a new project board to use templates * improve instructions for updating Zenodo * Apply suggestions from code review Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
* Update the development timeline for METplus version 6.2.0 and fix a minor error in the Release Guide insturctions for MET. * Removing accidental new file.
* clean up * per #3068, add utility function to download files and added unit tests to confirm it works as expected * return from function is uncompress extension is not present * remove unnecessary close() calls because with handles closing of file handles * Log error and return from init function if c_dict was not initialized properly. This allows create_c_dict function to return None in the child class wrappers (not parent classes) * per #3068, create basic components of DataIngest wrapper * Add argument to getraw function to skip replacement of double slash with single slash. The replacement was added to prevent errors with paths that contain double slashes in MET config files, but this causes issues with URLs. * clean up to remove unnecessary variable * Rename DATA_INGEST_<n>_URL/LOCAL_PATH to INPUT/OUTPUT_TEMPLATE to be consistent with other wrappers. Fix incorrect has_option config function name. Improve logging so it is clear that a file is currently being downloaded. Added skip-if-output-exists logic * added basic use case for new wrapper * Handle gz extension as gzip file. Refactor new functions to reduce cognitive complexity * add option to auto decompress files before writing to disk * add use case documentation * add new use case to run in automated test suite * added config options for INPUT/OUTPUT_DIR to be consistent with other wrappers -- added workaround for getdir function to prevent replacement of double slash in URLs * fixed typo * added documentation for config variables for new wrapper * turn off new use case before creating PR * remove test that takes a very long time to download file * removed deprecated config variable * fixed bug in enhancement to config getdir to properly handle skip of double slash handling * improve logging for readability * change lead sequence in example because it is not used * Per #3068, add unit tests for new DataIngest wrapper * fix typo * refactor automated test scripts to reduce duplication in logic to run commands * improve logging so it is more clear that a diff test failed because it is likely that the truth data does not exist yet (e.g. for new use cases) and exit 2 to match return code for a use case that contains actual diffs in the output
… first bugfix release. (#3093)
…d EnsembleStat wrappers (#3092)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updating files, truth data for v6.2
Created by @j-opatz