* fixed TCPairs unit tests to handle default value for match_points * bugfix #1691 remove whitespace from output file paths (#1721) * added variables that are needed for use case and were accidentally removed * log error and increment error count if required lat/lon bound variables are not set instad of calling sys.exit * skip metplus_final.conf in check for new output to prevent false alarm diffs * Update index.rst replace METdatadb with METdataio * Update metdatadb.rst Replace metdatadb with metdataio * renamed from metdatadb * replacing all other instances of metdatdb/METdatadb with metdataio/METdataio * after renaming directory from metdatadb to metdataio * replace METdatadb with METdataio * copied and replaced with metdataio filenames * replaced with metdataio version * Update the Release Guide while creating the MET-11.0.0-beta2 development release. Automatic Zenodo updates for each MET release have been DISABLED since they were being created for each development release. Revise the MET instructions to manually update the MET version on Zenodo only for official releases, not development or bugfix releases. * Changed version number * Added release notes * Update the labels scripts by adding 1 new one and renaming metdatadb to metdataio. * updated pandas version used for METdataio use cases to use 1.2.3 instead of 1.4.3 * update version to show development towards beta3 release * Changed METdatadb references to METdataio * Adding modulefiles for installation on various machines. (#1750) * Feature 1746 recreate release (#1751) * Adding instructions for recreating and existing release * Reorder the step's based on suggestions from @georgemccabe * Feature #1489 plot point obs wrapper (#1753) * Fixing typo * Feature #1657 clean up directory creation (#1756) * Feature update labels (#1765) * Modifications for METbaseimage * Update README.md Co-authored-by: johnhg <johnhg@ucar.edu> * feature dtcenter/METplus-Internal#20 add user ID to log output (#1771) * Per dtcenter/METplus-Internal#20, change log output to always include username and include user ID if it is available * Per dtcenter/METplus-Internal#20, fixed bug introduced from commit 458714c to develop where error is thrown if username cannot be obtained, which occurs in environment running GHA tests * Revert "Per dtcenter/METplus-Internal#20, fixed bug introduced from commit 458714c to develop where error is thrown if username cannot be obtained, which occurs in environment running GHA tests" This reverts commit fb2d093. * Revert "Per dtcenter/METplus-Internal#20, change log output to always include username and include user ID if it is available" This reverts commit 458714c. * per dtcenter/METplus-Internal#20, implemented working solution to user info logging * change file path based on change to MODE output from dtcenter/MET#2251 * fix Contributor's Guide GitHub Workflow page (#1774) * Feature dtcenter/METplus-Internal#32 add timestamp to final conf and file_lists (#1763) * Feature dtcenter/METplus-Internal#15 checksum for release (#1769) Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature dtcenter/METplus-Internal#32 change SCRUB_STAGING_DIR default and add documentation (#1784) * fix to allow wildcard characters in UserScript input templates, ci-run-all-diff * per #1787, allow NA instead of only integers for climo_mean/stdev hour_interval * per #1787, added test for setting NA for climo_mean hour_interval * Updating note format * update Python version to 3.8.6 * Revert "update Python version to 3.8.6" This reverts commit c1e6cbc. * added return value checking to error and exit a non-zero value if docker build or docker push fail * update instructions to find issues for a bugfix release (#1790) * Feature dtcenter/METplus-Internal#34 set the log time to UTC/GMT (#1793) * Feature #1330 MJO ENSO use case (#1794) Co-authored-by: cristianastan2 <cstan@gmu.edu> * Feature #1816 ensemble vx removal from EnsembleStat (#1820) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature #1816 rename cases and files (#1823) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Move files under internal_tests into internal/tests (#1818) * info for new use case * renamed picture of unrelated use case to properly display * Minor tweak to the release guide instructions for MET development releases. * Feature #1817 5.0.0-beta3 release (#1825) Co-authored-by: bikegeek <minnawin@ucar.edu> * Update VERSION * updated file name * updated MET file locations * explicitly set units for obs read via python embedding since PR dtcenter/MET#2288 no longer guesses units based on grib table units (which can be inaccurate of the actual data) * turn off precip use case group * Feature #1829 set unique ID in config (#1830) * fixed typo in use case setting that causes differences in the test output * Feature #1687 RuntimeFreq read only files needed for given run (#1841) * Feature 1847 release (#1850) * Per #1847, moved release notes to be its own chapter, reformatted component release notes, and added new section for upgrade instructions * Per #1847, added release-notes internal tag * Per #1847, fix error and reformat link * Per #1847, modified release instructions to add a section about upgrade instructions * Per #1847, modified release instructions to add a section about upgrade instructions * Per #1847, resolve doc warnings * Per #1847, updated instructions for clarity. * Per #1847, modified text and link. * Per #1847, modified wording and resolve error with note * Feature #1863 fix GitHub Actions warnings (#1864) * Feature #1874 prevent false diffs (#1875) * Feature 1842 StatAnalysis - multiple issues (#1869) * Bugfix #1884 develop PCPCombine {custom} in subtract method (#1887) * Feature 1852 use case physics tendency (#1881) * Issue #1852 initial work for the Planview plot * Issue #1852 planview plot for physics tendency * issue #1852 cleared away unused/unneccessary code * Issue #1852 update script command * Issue #1852 first draft, doesn't have correct content * Issue #1852 updated name of output plot to include short_range in name * Issue #1852 renamed to include fcstOnly label * Issue #1852 initial commit * Issue #1852 * Issue #1852 * Issue #1852 initial commit * issue #1852 vertical cross section * Issue 1852 clean up unused lines * Issue #1852 initial commit * Issue #1852 * Issue #1852 updates * Issue #1852 fixed documentation warnings, but some problems remain * fixing line length for chapter and section headers and one letter capitalization * cleaning up line spacing and trying to fix a critical error. * trying to fix critical error again * Issue #1852 more description for data and dependency on METplotpy with additional instructions * Issue #1852 update location of input data to single directory * Removing duplicate files with a misformatted name that are not needed * Per #1852, making changes to formatting to eliminate errors * Issue #1852 remove redundant yaml config file * Issue #1852 update PYTHONPATH information * Issue #1852 fixed conflicts * Issue #1852 update path to shapefile * Issue #1852 fixed copy and paste error * Issue #1852 add three use case entries: Planview, VerticalProfile, and VerticalCrossSection for the short range Physics tendency use case * Issue #1852 add latest short range tests to top, move down list when everything passes * Update use_case_groups.json move short_range use cases #10-12 to below the short range test for #9 * Issue #1852 clean up conf files by providing variables for the input netCDF files and for the planview plot, allow user to indicate pressure levels * Issue #1852 Indicate external Python packages * Issue #1852 added external dependency section * removed env vars that don't need to be set -- METPLOTPY_BASE is set by the automated test conf (.github/parm/test_settings.conf) and PYTHONPATH doesn't need to be set because METplotpy is installed in Python environment for use case tests * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py Fixed one typo * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_VerticalCrossSection.py Fixed a typo * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_VerticalProfile.py Same typo, same fix * Issue #1852 remove extraneous entry for USER_SCRIPT_COMMAND, a hold-over from the original implementation which doesn't allow the user to easily specify the FV3 history file and grid spec netCDF file * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_VerticalProfile.conf Removed the incorrect USER_SCRIPT_COMMAND in previous commit. Reinstated the correct version and removed the incorrect command. * update html file from Planview to VerticalProfile Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> * Feature #1893 StatAnalysis custom looping (#1894) * Feature dtcenter/METplus-Internal#24 test failure (#1895) * updated version for creating release (#1897) * update release number for development towards beta5 * Adding one sentence to about daily follow up on discussions. * initial changes for new use case * updated paths, documentation * updated automated testing files * Feature #344 rearrange utils (#1900) * first additions for new use case, corrected documentation for old use case * Move Docker scripts to be consistent with other METplus components (#1907) * added picture * Correct spelling mistake * handle if input to format_thresh is already a list * Revert "handle if input to format_thresh is already a list" This reverts commit d329275. * handle if input to format_thresh is already a list * per #1910, added time util function to add field info to time info dictionary and updated wrapper logic to use this function to add this info instead of passing field info (var_info) to CommandBuilder.find_data to get level info, ci-run-all-diff * change logic to preserve full level string for [fcst/obs]_level so SeriesAnalysis cases build the same file names, but parse out time info for setting generic level for string sub to preserve previous behavior * try setting all input values in output dictionary, ci-run-all-diff * removed commented code * add quotes around input files to plot_point_obs command to allow support for Python Embedding as input * put all arguments on single line * remove commented code * per #1910, added unit tests to ensure that level is set properly in the forecast input template * added script to create geovista conda environment via Docker to use in the automated use case tests * Feature 1561 use case ugrid python embedding (#1927) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 1453 use case iodav2 python embedding (#1930) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * fix permissions for script that were somehow changed to be not executable * Bugfix #1939 develop - failure reading obs when zipped file also exists (#1941) * Feature #1819 alert users if updates to EnsembleStat use cases are needed (#1945) Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * fixed underline that did not match title * Feature #1898 tc new options (#1932) close undefined * fixed broken doc util logic from bad PR merge * Feature #1569 EnsembleStat -ens_mean argument (#1952) * Feature #1953 seeps config in GridStat and PointStat (#1954) Co-authored-by: johnhg <johnhg@ucar.edu> * Feature #1566 update Python version to 3.8.6 (#1958) * Feature #1956 5.0.0-beta4 release (#1963) * Update VERSION * added warning in release notes that location of METplus Dockerfile has changed * combined release notes for all betas for the v5.0.0 release * updated tags for each METplus component to match the upcoming release * Feature #1885 gh_projects (#1973) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Switch from double to single quotes. * Update formatting of the GitHub project filters so that they can be copy/pasted directly from RTD into the GitHub project filter. * Feature #1953 PointStat output_flag.seeps and output_flag.seeps_mpr (#1976) * Feature #1711, #1783, #1814, and #1835 Documentation - Multiple files, PCPCombine constant init, Docker instructions, and MET links (#1967) Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * run diff logic on push for develop and main_v* branches, fix logic to properly run all use cases * fixed name of external * added Bugfix: to default title of new bugfix issue * added space after New Use Case: * changed references to METdatadb to METdataio that were missed * Update overview.rst Removed statsmodels from the table, this package is no longer a requirement. The necessary functionality was pulled out of the statsmodels repository and incorporated into the METcalcpy repository * Update overview.rst METdataio *must* only use pandas 1.2.3. METdataio also requires pyyaml. * Update overview.rst METdatio uses pytest to test aspects of the METreformat module * Update rotate_authorship.rst Explicitly add the step to commit and push changes to the file containing the author list. * Update overview.rst Added METdataio for lxml, numpy, pandas, and pymysql * Update finalize_release_on_github_official.rst added branch protection rules step and the interface to Github was modified, you need to click on the stacked '->' icon to switch default branches * CI: replace / with _ to prevent errors creating docker tags for tests * Update overview.rst included METdataio for python 3.8.6+ requirement * Update overview.rst psutil no longer needed in METplotpy and METcalcpy * Update overview.rst METcalcpy no longer requires imutils and allow versions 0.5.3 and above * Update overview.rst cleaned up formatting for imutils * change asterisk to double asterisk to match branches that include forward slash / * update version of certifi based on dependabot auto-generated PR #1982 * Feature 1819 format doc (#1981) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update use case keywords (#1984) * Feature #1980 Use the main_vX.Y of METplus components for main METplus tests (#1985) * Feature #1986 v5.0.0 release (#1987) Closes #1986 * updated version for development towards beta1 of 5.1.0 * allow rest of workflow to run even if job to update data volumes fails in case server is down * updated version for action to update input data volumes and added max number of DockerHub tag pages to search to 15 * updated release notes for official METplus release to untar the sample data for next version directory * Update release guide to create the input test data directory for official MET releases. * Follow the METplus naming conventions for updating the the DTC web server data. * install library needed for geovista package, remove editable flag so geovista code will be added directly to python instead of reading it from the source code location, ci-skip-all * Feature 685 log updates (#1992) * add script to create conda environment with MetPy 1.4 for SWPC use case * Bugfix develop Fix broken documentation links (#2004) * fixed bug to set environment variables in the RegridDataPlane environment instead of the PyEmbedIngest environment * Feature #2015 modulefiles (#2024) * Adding modulefile for WCOSS2 * Updated references of 4.1.1 to 5.0.0 * Feature #2005 updates to create v5.1 conda environments that use debian10 and Python 3.10.4 (#2043) * update versions to fix bugs for METplotpy/calcpy use cases * Bugfix #2026 develop StatAnalysis looping (#2028) * Feature #2022 update python version to 3.10.4 for repo files (#2047) * Feature #2022 Use Debian 10 / Python 3.10.4 in automated tests (#2050) * add use case group name to output path so that output and error logs will contain use case category to more easily see which case was run * remove -group0 from group names for output path * add manual workflow to update truth data * ci: set gh token on checkout to enable push, ci-skip-all * Set user name/email via git config to allow push * add token to checkout * Feature #2049 Multiple interp.type.method/width (#2051) * added missing argument to script if building envs locally * get main_vX.Y truth data if running on main_vX.Y branch * Updating README.md for label creation and adding bold to common_labels.txt * Updated color for bold label * Feature #2054 consensus.write_members in TCPairs (#2057) * Create LICENSE.md (#2061) * per #2006, fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) * Feature 2065 v5.1.0 beta1 (#2066) * updated version number * added release notes * Update docs/Users_Guide/release-notes.rst Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * updated version with -dev * adding sphinx design to 3 files so dropdown menus will work in the release notes. * fix version numbers for python requirements for building documentation * fixed tab vs spaces * Bugfix #2070 var list numeric order (#2072) * Adding files for jet and hera * added commented process list that was accidentally removed -- this is used if the pre-processing steps are needed * Bugfix #2087 develop docs_pdf (#2091) * Feature 2076 release dropdown menu (#2083) * test dropdowns * Update release-notes.rst Attempting to resolve WARNING * testing dropdowns * adding dropdown menus * testing dropdown remove from warning section * dropdown testing * dropdown menu tests * dropdown menus after the warnings section, the dropdown menu only has one space instead of the normal 2 spaces. If it has 2 spaces, it gets sucked into the warning section. * spaces with warning * no spaces for warning * Update release-notes.rst warning note has to be indented 1 space. Originally it was no spaces but then all of the dropdown menus end up in the warning message. If it's 2 spaces, as is normal, then the warning is indented, which is unnecessary. 1 space is slightly indented and keeps the dropdown menus separate. --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Bugfix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) * Feature 1516 use case mvmode (#2094) Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see urllib3/urllib3#2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update github_workflow.rst --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: bikegeek <minnawin@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu>

Write Python code to handle string template substitution #10

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions