Update develop-ref after dtcenter/MET#3202 #3082
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe
* per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe822. * per #2567, fix bug that caused file window logic to fail
* Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg
…ily disabled land_surface:0 until we can resolve the differences
* Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting
…to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update
…tead merge changes from -ref into the update branch
… Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- #2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward
* prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM
* Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
* refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc
* change wording of error log to satisfy test
* set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup
* refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors
* refactor duplicate code into function to satisfy SonarQube
* removed unused variable
* use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method
* fix typo in key
* suppress warnings when files aren't found because it is expected
* formatting to be consistent in doc string
* per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs
* add documentation blocks for new functions, ci-run-all-diff
* use pytest fixture instead of local function
* add support for setting -vld_thresh argument
* per #2578, added documentation and tests for setting -input_thresh and -vld_thresh
* moved verbosity argument to end of command to more easily change it when debugging
* refactor to reduce cognitive complexity to satisfy SonarQube
* update usage statement to include METplus version number
* add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired
Update METplus-6.0.0 development schedule.
* Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification
…command to upload asset
* Per #2656, update the read_iodav2_mpr.py scripts to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Per #2656, update the ugrid_lfric_mpr.py to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns.
* per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting
* Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
* added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu>
* Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use case config gile to remove unused items. * Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min. * Removes unused functions from forecast Python embedding scripts. * Switches to F strings and wraps some print statements in DEBUG flag. * Switches to F strings and adds script name to print statements. * Adds space around print statements to aid log file inspection. * Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file. * Adds controlling DEBUG level for observation Python embedding scripts. * Updates documentation URL in config file. * Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA. * Updates for sample data and running the use case in GHA. * Typo in the JSON file. * Changes the MET_PYTHON_DIR to the installed location rather than the source code location. * Adds user script to create graphic from PointStat MPR output. * Adds use case gallery image and updates observation dataset info in documentation. * Fixes path to config file include in documentation. * Adds new user script for plotting to the documentation file. * Updates list of expected output files to include UserScript outpout and PB2NC output. * Changes to eliminate code smells. * Finalizes forecast dataset description and scientific objective. * Adds info on additional Python packages required for Python embedding and UserScripts. * Removes extra #. * Removes uneccesary imports in user script. * Adds more clarity that METcalcpy is a required Python package for Python embedding. * Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts. * Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3. * Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts).
… periodically to fix any security vulnerabilities in upstream images
Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu>
* Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extension * add more information for use case doc * turn off use case for PR * refactor map functions script to reduce cognitive complexity for SonarQube * update scientific objective with input from @jaredalee * update use case to reference environment variables in reformatted config file instead of using relative paths and changing directories since changes needed to support env vars in the reformatted are available in develop now * test that changes to use case did not break anything
* Initial config file for this use case. * Adds ismn as an acceptable format for ASCI2NC_INPUT_FORMAT. * Latest param file changes. * Documentation file for this use case. * Updates documentation URL in config file. * Updated conf file that produces matched pairs for soil moisture from PointStat. * Updates to documentation for ISMN use case. * Adds UserScripts for plotting. * Latest conf file settings for use case. * Use case gallery image. * Adds summary using NEAREST for point_stat matched pairs. * Adds path to gallery image for use case. * Changes use case template to have the sphinx gallery line separate from the previous section to avoid having the actual sphinx_gallery_path appear in the rendered HTML documentation. Still need to fix use cases that don't have this change. * Per discussion with @georgemccabe and experience using the StatAnalysis Wrapper for the ISMN use case, moving two stat-analysis wrapper config items that were listed as required to the optional section since it worked without setting them. Presumably they may be required based on how a user is running stat-analysis or the stat-analysis wrapper, but they are not always required. * Per discussion with @georgemccabe, adds additional details for adding a new use case instructing the user to return the run setting to false in the use_case_groups.json file prior to merging the PR, which is the desired default setting for all use cases. * Minor change to use case documentation. * Initial work for adding ISMN to Verification Datasets guide. * Adds documentation for the ISMN dataset in the Verification Dataset Guide. * Adds VxDataISMN to use case documentation to link the use case to the verification datasets guide. * Adds new use case to testing suite after adding sample data. * Removes mention of RAL FTP site for sending sample data for a new use case, since that is not allowed anymore and generically refers to the GitHub Discussion describing the new process using Google Drive with a link to the Discussion. * Forgot a space.
* Update to MET 12.0.3 bugfix version. * Switch from using "secrets.DOCKER_USERNAME/secrets.DOCKER_PASSWORD" to using "dtcenter/secrets.DOCKER_TOKEN".
* Per #3054, enhance release-docker-images.yml to support CVE scanning, scheduled weekly runs for one or more versions, and update the logic for pushing X.Y-latest tags to Docker Hub. * Per #3054, update the Release Guide with instructions about updating the GHA release workflows. * Per #3054, fix workflow typo. * Per #3054, update release guide formatting. * Per #3054, add CMD_LOGFILE logic to the time_command function. * Per #3054, refine release guide instructions. * Per #3054, refine release guide instructions. * Per #3054, update the versions in the release-docker-images.yml workflow for version 6.1.0. * Per #3054, update to plotly 6.1.1 and kaleido 1.0.0 versions. * Per #3054, revert changes to python package versions since those do not actually impact the GHA testing environment. * Per #3054, updating plotly version to 6.1.1 and python-kaledio version to 1.0.0 in the documentation and internal scripts. Hoping that will enable the 4 failing METplus use cases to succeed.
…ETplotpy use cases
…e MET/METviewer/METexpress store downloads in a separate website location than the other components. (#3074)
…3075) * update instructions to create a new project board to use templates * improve instructions for updating Zenodo * Apply suggestions from code review Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
11 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
dtcenter/MET#3202 included changes relating to madis2nc
Created by @georgemccabe