Skip to content

Conversation

github-actions[bot]
Copy link
Contributor

Modifies existing STAT-Analysis output file.
Created by @JohnHalleyGotway

Howard Soh and others added 30 commits February 22, 2024 21:16
…ullptr

Feature 2673 sonarqube beta4 nullptr
…eturn

Feature 2673 sonarqube beta4 return
…amespace

Feature 2673 sonarqube beta3 namespace
* Unrelated to #2583, fix typo in code comments.

* Per #2583, add hooks write 3 new ECNT columns for observation error data.

* Per #2583, make error messages about mis-matched array lengths more informative.

* Per #2583, switch to more concise variable naming conventions of ign_oerr_cnv, ign_oerr_cor, and dawid_seb.

* Per #2583, fix typo to enable compilation

* Per #2583, define the 5 new ECNT column names.

* Per #2583, add 5 new columns to the ECNT table in the Ensemble-Stat chapter

* Per #2583, update stat_columns.cc to write these 5 new ECNT columns

* Per #2583, update ECNTInfo class to compute the 5 new ECNT statistics.

* Per #2583, update stat-analysis to parse the 5 new ECNT columns.

* Per #2583, update aggregate_stat logic for 5 new ECNT columns.

* Per #2583, update PairDataEnsemble logic for 5 new ECNT columns

* Per #2583, update vx_statistics library with obs_error handling logic for the 5 new ECNT columns

* Per #2583, changes to make it compile

* Per #2583, changes to make it compile

* Per #2583, switch to a consistent ECNT column naming convention with OERR at the end. Using IGN_CONV_OERR and IGN_CORR_OERR.

* Per #2583, define ObsErrorEntry::variance() with a call to the dist_var() utility function.

* Per #2583, update PairDataEnsemble::compute_pair_vals() to compute the 5 new stats with the correct inputs.

* Per #2583, add DEBUG(10) log messages about computing these new stats.

* Per #2583, update Stat-Analysis to compute these 5 new stats from the ORANK line type.

* Per #2583, whitespace and comments.

* Per #2583, update the User's Guide.

* Per #2583, remove the DS_ADD_OERR and DS_MULT_OERR ECNT columns and rename DS_OERR as DSS, since observation error is not actually involved in its computation.

* Per #2583, minor update to Appendix C

* Per #2583, rename ECNT line type statistic DSS to IDSS.

* Per #2583, fix a couple of typos

* Per #2583, more error checking.

* Per #2583, remove the ECNT IDSS column since its just 2*pi*IGN, the existing ignorance score, and only provides meaningful information when combined with the other Dawid-Sebastiani statistics that have already been removed.

* Per #2583, add Eric's documentation of these new stats to Appendix C. Along the way, update the DOI links in the references based on this APA style guide: https://apastyle.apa.org/style-grammar-guidelines/references/dois-urls#:~:text=Include%20a%20DOI%20for%20all,URL%2C%20include%20only%20the%20DOI.

* Per #2583, fix new equations with embedded underscores for PDF by defining both html and pdf formatting options.

* Per #2583, update the ign_conv_oerr equation to include a 2
*pi multiplier for consistency with the existing ignorance score. Also, fix the documented equations.

* Per #2583, remove log file that was inadvertently added on this branch.

* Per #2583, simplify ObsErrorEntry::variance() implementation. For the distribution type of NONE, return a variance of 0.0 rather than bad data, as discussed with @michelleharrold and @JeffBeck-NOAA on 3/8/2024.

---------

Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
JohnHalleyGotway and others added 24 commits May 21, 2025 10:46
* Per #3155, update the version and release notes for version 12.1.0-rc1.

* Per #3155, add upgrade instructions.

* Per #3157, tweak bugfix wording.

* Per #3157, improve formatting of configuration options.

* Per #3157, remove empty METbaseimage testing environment section from RC1 releates notes.
* Per #3165, correct two poorly formed OpenMP pragma statements.

* Per #3165, add missing array include to ensemble_stat, point_stat, and plot_point_obs that's appearently needed when compiling with the Intel OneAPI compilers and with the --disable-opemp configuration option.

* Per #3165, move 'using namespace std' after the includes in the all the test code and enum_parser.yy to avoid Intel OneAPI compilation warnings about it being implicitly defined.

* Per #3165, only compile the kdtree test programs when ENABLE_UGRID is true.

* Per #3165, only compile the kdtree test programs when ENABLE_UGRID is true.

* Per #3165, move 'using namespace std' after the includes in the all the test code.

* Per #3165, add DISABLE_OPENMP option to compilation script

* Per #3165, updating beta2 reference to rc1

* Per #3165, fix typo in internal/test_util/libcode/vx_grid/Makefile.am

* Per #3165, removing typo in acorn config file

* Per #3165, Test whether disabling custom reductions from the vx_series_data library enables MET to compile on WCOSS2 with Intel classic compilers.

---------

Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>
* Per #3179, update the logic to not search the parent when parsing file_type from climatology dictionaries. (#3180)

* Per #3179, modify the climo_name file_type logic to parse it from the top-level climo dictionary rather than parsing it separately for each field array entry.

---------

Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
* #3164 Removed duplicated code

* #3164 Added more API to handle the slice by value

* Merged fixes for the SonarQube

* #3164 Fix the SonarQUbe findings

* #3164 Corrected const declarations

* #3164 Corrected const declarations

* #3164 Fixes for SonarQube findings

* #3164 Fixes for SonarQube findings

* #3164 Fixes for SonarQube findings

* #3164 Fixes for SonarQube findings: const

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* #3164 Suuport @lower-@upper notation

* #3164 Renamed error_code_bad_offset to error_code_out_of_index

* Bugfix 3164 slice range sonarqube (#3173)

* #3164 Fix the SonarQUbe findings

* #3164 Corrected const declarations

* #3164 Corrected const declarations

* #3164 Fixes for SonarQube findings

* #3164 Fixes for SonarQube findings

* #3164 Fixes for SonarQube findings

* #3164 Fixes for SonarQube findings: const

* Renamed error_code_bad_offset to error_code_out_of_index

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* #3164 Initialize cur_time_index and cur_z_index

* #3164 Changed offset 0 to time string. Changed two @ notation level

* #3164 Corrected typo

* #3164 SonarQube code smell

* #3164 SonarQube code smell

* #3164 CHnaged warning to error for bad offset for Z

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>
Fix typo in code comment. No real code changes.
Make the setting of SQLITE_INCLUDE_DIR and SQLITE_LIB_DIR more portable.
Revert the development.docker environment back to setting MAKE_ARGS="-j 5" that was used for version 12.0.0. Use unlimited "-j" causes an error when building ecKit, likely consuming too many resources in the Docker image.
* Updated note in UGRID documentation explaining Point-Stat does not need regridding.

* utilized secondary opinion on language update

* Update language to remove ambiguity

Co-authored-by: John Halley Gotway <johnhg@ucar.edu>

---------

Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
* updates to test scripts to replace comp_dir.R with comp_dir.py

* fixed saving of diff files (sort of)

* fixed how diff files are saved to diff directory; general tidying

* updated .gitignore to only ignore top level bin dir, not other files/dirs containig bin

* updated how path for METplus module is supplied

* changed test_regression.sh to call comp_dir.py directly instead of wrapper script and removed comp_dir.sh--wrapper not necessary

* updating run_diff_tests.sh in GH jobs to use metplus diff

* updating run_diff_tests.sh in GH jobs to use metplus diff

* updated github diff scripts to run metplus-envs:diff container

* minor fix in docker command

* more fixes to run diff tests in docker

* more fixes to run diff tests in docker

* more fixes to run diff tests in docker

* updated copying diff files

* remove ref to copy_diff_files.log

* added log statements for copying diff files

* added print statements

* added time tracking to comp_dir, changed metplus branch for testing

* trivial change to logging output

* updated to use develop branch of metplus for tests

* tidying & attempt at programmatic way to get diff docker version

* tidying

* tidying, continued

* updated METPLUS_DIR env var for nightly build tests

---------

Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu>
…he PATH so that the comp_dir.py script uses the intended Python version.
* Per dtcenter/METbaseimage#32, port over changes from main_v12.0 to develop, but update the schedule to rebuild images weekly rather than daily.

* Better comment
* Per #3148, add new is_directory() utility function.

* Per #3148, add new is_ascii_file_list() utility function.

* Per #3148, add new is_ascii_file_list() utility function.

* Per #3148, add new -inputrx command line option and add logic to parse input files on the command, from a file list, or by searching a directory.

* Per #3148, update the get_filenames() functions to check input regular expressions directly rather than a filename prefix and suffix that it converts to regular expressions.

* Per #3148, update all calls to get_filenames() to pass regular expressions as arguments rather then file prefix and suffix strings.

* Per #3148, use more consistent logging levels.

* Per #3148, update ascii2nc unit tests to demonstrate providing inputs with file lists or searching input directories.

* Per #3148, update the ascii2nc documentaiton.

* Per #3148, refine the ascii2nc usage statement, refine log messages, and make sure python inputs work.

* Per #3148, update ascii2nc to support multiple python inputs all specified as quoted strings on the command line, rather than only one.

* Per #3148, correct the usage statement in the User's Guide.

* Per #3148, fix typo in ascii2nc unit test.

* Per #3148, define atcf_suffix and atcf_reg_exp in one spot in the vx_tc_util library instead of redefining it in each TC application.

* Per #3148, address easy SonarQube code smells.g

* Per #3148, correct logic in get_input_files()... had an if that should be else if.

* Per #3148, whitespace diffs.

* Per #3148, update the ascii2nc error-handling logic. Rather than a bad or missing filename causing ascii2nc to error out completely, process as many input files as it can and print warning messages about problematic inputs.

* Not released to #3148, but coordinated with Howard Soh to update the formatting of the method_name strings that appear in log messages for consistency.

* Per #3148, remove unneeded exclamation points

* Per #3148, for file handlers, generally, change Error log messages to Warning since the top level file handler no longer exits with bad status when failing to read an input file. Instead, a warning is printed since processing will continue to the next file.

* Per #3148, add DEBUG(2) log message to list the number of successful input file reads.

* Per #3148, update ascii2nc to print the -format option being applied when the user has not provided one on the command line.

---------

Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
* Per #3198, port over CVE scanning logic from METbaseimage to MET.

* Per #3198, use SOURCE_BRANCH instead of GITHUB_NAME for the scan_docker_image.sh script.

* Fix typo
* Unrelated to #3206, increase the METbaseimage version number from v3.4 to v3.4.1 so that the resulting Docker images contain 0 critical vulnerabilities.

* Per #3206, update met_regrid_area_weighted() function to fix bug when OMP_NUM_THREADS is set very large. Need to use a reduction for vectors of doubles to prevent two threads from updating the same to_data value at the same time.

* Per #3206, add comment.
* Fix jobstring to write -out_wind_logic correctly instead of reporting the value for cnt_logic instead.

* Restore logic in aggr_mpr_wind_lines() removed for #2395 when STAT-Analysis aggregates MPR lines to compute WDIR output. In main_v11.0, that job automatically removes 0 vectors when computing ROW_MEAN_WDIR output.

* Per #3210, update the revision history

* Per #3210, update the WDIR documentation

* Per #3210, add one more WDIR job to STAT-Analysis using all inputs, not just the non-zero MPR vectors.

* Unrelated to #3210, but noted by @CPKalb, note that MET can read raw WRF output.

* Per #3210, solve one additional bug in the MPR -> VCNT aggregation. Since the local v_info object was declared outside the loop, when d_diff is bad data, the values from the previous iteration were being using. This cause the TOTAL_DIR value to be reported as the TOTAL value and not properly skip over the zero length vectors as it should... and as Point-Stat does.

---------

Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
@github-project-automation github-project-automation bot moved this to 🩺 Needs Triage in METplus-6.2 Development Jul 24, 2025
@JohnHalleyGotway JohnHalleyGotway added this to the MET-12.2.0 milestone Jul 24, 2025
@JohnHalleyGotway JohnHalleyGotway moved this from 🩺 Needs Triage to 🏁 Done in METplus-6.2 Development Jul 24, 2025
@JohnHalleyGotway JohnHalleyGotway merged commit 78f01f6 into develop-ref Jul 24, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 🏁 Done
Development

Successfully merging this pull request may close these issues.

Bugfix: Fix STAT-Analysis handling of zero-length vectors when aggregating MPR inputs and writing WDIR or VCNT outputs