Update develop-ref after #3185 #3223
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…eturn Feature 2673 sonarqube beta4 return
…qube_beta3_namespace
…amespace Feature 2673 sonarqube beta3 namespace
#2673 Changed literal 1 to the boolean value, true
* Unrelated to #2583, fix typo in code comments. * Per #2583, add hooks write 3 new ECNT columns for observation error data. * Per #2583, make error messages about mis-matched array lengths more informative. * Per #2583, switch to more concise variable naming conventions of ign_oerr_cnv, ign_oerr_cor, and dawid_seb. * Per #2583, fix typo to enable compilation * Per #2583, define the 5 new ECNT column names. * Per #2583, add 5 new columns to the ECNT table in the Ensemble-Stat chapter * Per #2583, update stat_columns.cc to write these 5 new ECNT columns * Per #2583, update ECNTInfo class to compute the 5 new ECNT statistics. * Per #2583, update stat-analysis to parse the 5 new ECNT columns. * Per #2583, update aggregate_stat logic for 5 new ECNT columns. * Per #2583, update PairDataEnsemble logic for 5 new ECNT columns * Per #2583, update vx_statistics library with obs_error handling logic for the 5 new ECNT columns * Per #2583, changes to make it compile * Per #2583, changes to make it compile * Per #2583, switch to a consistent ECNT column naming convention with OERR at the end. Using IGN_CONV_OERR and IGN_CORR_OERR. * Per #2583, define ObsErrorEntry::variance() with a call to the dist_var() utility function. * Per #2583, update PairDataEnsemble::compute_pair_vals() to compute the 5 new stats with the correct inputs. * Per #2583, add DEBUG(10) log messages about computing these new stats. * Per #2583, update Stat-Analysis to compute these 5 new stats from the ORANK line type. * Per #2583, whitespace and comments. * Per #2583, update the User's Guide. * Per #2583, remove the DS_ADD_OERR and DS_MULT_OERR ECNT columns and rename DS_OERR as DSS, since observation error is not actually involved in its computation. * Per #2583, minor update to Appendix C * Per #2583, rename ECNT line type statistic DSS to IDSS. * Per #2583, fix a couple of typos * Per #2583, more error checking. * Per #2583, remove the ECNT IDSS column since its just 2*pi*IGN, the existing ignorance score, and only provides meaningful information when combined with the other Dawid-Sebastiani statistics that have already been removed. * Per #2583, add Eric's documentation of these new stats to Appendix C. Along the way, update the DOI links in the references based on this APA style guide: https://apastyle.apa.org/style-grammar-guidelines/references/dois-urls#:~:text=Include%20a%20DOI%20for%20all,URL%2C%20include%20only%20the%20DOI. * Per #2583, fix new equations with embedded underscores for PDF by defining both html and pdf formatting options. * Per #2583, update the ign_conv_oerr equation to include a 2 *pi multiplier for consistency with the existing ignorance score. Also, fix the documented equations. * Per #2583, remove log file that was inadvertently added on this branch. * Per #2583, simplify ObsErrorEntry::variance() implementation. For the distribution type of NONE, return a variance of 0.0 rather than bad data, as discussed with @michelleharrold and @JeffBeck-NOAA on 3/8/2024. --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
* Unrelated to #2583, fix typo in code comments. * Per #2583, add hooks write 3 new ECNT columns for observation error data. * Per #2583, make error messages about mis-matched array lengths more informative. * Per #2583, switch to more concise variable naming conventions of ign_oerr_cnv, ign_oerr_cor, and dawid_seb. * Per #2583, fix typo to enable compilation * Per #2583, define the 5 new ECNT column names. * Per #2583, add 5 new columns to the ECNT table in the Ensemble-Stat chapter * Per #2583, update stat_columns.cc to write these 5 new ECNT columns * Per #2583, update ECNTInfo class to compute the 5 new ECNT statistics. * Per #2583, update stat-analysis to parse the 5 new ECNT columns. * Per #2583, update aggregate_stat logic for 5 new ECNT columns. * Per #2583, update PairDataEnsemble logic for 5 new ECNT columns * Per #2583, update vx_statistics library with obs_error handling logic for the 5 new ECNT columns * Per #2583, changes to make it compile * Per #2583, changes to make it compile * Per #2583, switch to a consistent ECNT column naming convention with OERR at the end. Using IGN_CONV_OERR and IGN_CORR_OERR. * Per #2583, define ObsErrorEntry::variance() with a call to the dist_var() utility function. * Per #2583, update PairDataEnsemble::compute_pair_vals() to compute the 5 new stats with the correct inputs. * Per #2583, add DEBUG(10) log messages about computing these new stats. * Per #2583, update Stat-Analysis to compute these 5 new stats from the ORANK line type. * Per #2583, whitespace and comments. * Per #2583, update the User's Guide. * Per #2583, remove the DS_ADD_OERR and DS_MULT_OERR ECNT columns and rename DS_OERR as DSS, since observation error is not actually involved in its computation. * Per #2583, minor update to Appendix C * Per #2583, rename ECNT line type statistic DSS to IDSS. * Per #2583, fix a couple of typos * Per #2583, more error checking. * Per #2583, remove the ECNT IDSS column since its just 2*pi*IGN, the existing ignorance score, and only provides meaningful information when combined with the other Dawid-Sebastiani statistics that have already been removed. * Per #2583, add Eric's documentation of these new stats to Appendix C. Along the way, update the DOI links in the references based on this APA style guide: https://apastyle.apa.org/style-grammar-guidelines/references/dois-urls#:~:text=Include%20a%20DOI%20for%20all,URL%2C%20include%20only%20the%20DOI. * Per #2583, fix new equations with embedded underscores for PDF by defining both html and pdf formatting options. * Per #2583, update the ign_conv_oerr equation to include a 2 *pi multiplier for consistency with the existing ignorance score. Also, fix the documented equations. * Per #2583, remove log file that was inadvertently added on this branch. * Per #2583, simplify ObsErrorEntry::variance() implementation. For the distribution type of NONE, return a variance of 0.0 rather than bad data, as discussed with @michelleharrold and @JeffBeck-NOAA on 3/8/2024. * Per #2583, updates to ensemble-stat.rst recommended by @michelleharrold and @JeffBeck-NOAA. * Per #2583, implement changes to the IGN_CORR_OERR corrected as directed by @ericgilleland. --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
… impacts to existing METplus Use Cases.
* Per #2833, fix n-1 bug when defining the azimuth delta for range/azimuth grids. * Per #2833, when definng TcrmwData:range_max_km, divide by n_range - 1 since the range values start at 0. * Per #2833, remove max_range_km from the TC-RMW config file. Set the default rmw_scale to NA so that its not used by default. And update the documentation. Still actually need to make the logic of the code work as it should. * Per #2833, update tc_rmw to define the range as either a function of rmw or using explicit spacing in km. * Per #2833, update the TCRMW Config files to remove the max_range_km entry, and update the unit test for one call to use RMW ranges and the other to use ranges defined in kilometers. * Per #2833, just correct code comments. * Per #2833, divide by n - 1 when computing the range delta, rather than n. * Per #2833, correct the handling of the maximum range in the tc-rmw tool. For fixed delta km, need to define the max range when setting up the grid at the beginning. --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
* #3164 Removed duplicated code * #3164 Added more API to handle the slice by value * Merged fixes for the SonarQube * #3164 Fix the SonarQUbe findings * #3164 Corrected const declarations * #3164 Corrected const declarations * #3164 Fixes for SonarQube findings * #3164 Fixes for SonarQube findings * #3164 Fixes for SonarQube findings * #3164 Fixes for SonarQube findings: const --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> * #3164 Suuport @lower-@upper notation * #3164 Renamed error_code_bad_offset to error_code_out_of_index * Bugfix 3164 slice range sonarqube (#3173) * #3164 Fix the SonarQUbe findings * #3164 Corrected const declarations * #3164 Corrected const declarations * #3164 Fixes for SonarQube findings * #3164 Fixes for SonarQube findings * #3164 Fixes for SonarQube findings * #3164 Fixes for SonarQube findings: const * Renamed error_code_bad_offset to error_code_out_of_index --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> * #3164 Initialize cur_time_index and cur_z_index * #3164 Changed offset 0 to time string. Changed two @ notation level * #3164 Corrected typo * #3164 SonarQube code smell * #3164 SonarQube code smell * #3164 CHnaged warning to error for bad offset for Z --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>
Fix typo in code comment. No real code changes.
Make the setting of SQLITE_INCLUDE_DIR and SQLITE_LIB_DIR more portable.
Revert the development.docker environment back to setting MAKE_ARGS="-j 5" that was used for version 12.0.0. Use unlimited "-j" causes an error when building ecKit, likely consuming too many resources in the Docker image.
* Updated note in UGRID documentation explaining Point-Stat does not need regridding. * utilized secondary opinion on language update * Update language to remove ambiguity Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
* updates to test scripts to replace comp_dir.R with comp_dir.py * fixed saving of diff files (sort of) * fixed how diff files are saved to diff directory; general tidying * updated .gitignore to only ignore top level bin dir, not other files/dirs containig bin * updated how path for METplus module is supplied * changed test_regression.sh to call comp_dir.py directly instead of wrapper script and removed comp_dir.sh--wrapper not necessary * updating run_diff_tests.sh in GH jobs to use metplus diff * updating run_diff_tests.sh in GH jobs to use metplus diff * updated github diff scripts to run metplus-envs:diff container * minor fix in docker command * more fixes to run diff tests in docker * more fixes to run diff tests in docker * more fixes to run diff tests in docker * updated copying diff files * remove ref to copy_diff_files.log * added log statements for copying diff files * added print statements * added time tracking to comp_dir, changed metplus branch for testing * trivial change to logging output * updated to use develop branch of metplus for tests * tidying & attempt at programmatic way to get diff docker version * tidying * tidying, continued * updated METPLUS_DIR env var for nightly build tests --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu>
…ree rather than /home/met_test.
…he PATH so that the comp_dir.py script uses the intended Python version.
* Per dtcenter/METbaseimage#32, port over changes from main_v12.0 to develop, but update the schedule to rebuild images weekly rather than daily. * Better comment
* Per #3148, add new is_directory() utility function. * Per #3148, add new is_ascii_file_list() utility function. * Per #3148, add new is_ascii_file_list() utility function. * Per #3148, add new -inputrx command line option and add logic to parse input files on the command, from a file list, or by searching a directory. * Per #3148, update the get_filenames() functions to check input regular expressions directly rather than a filename prefix and suffix that it converts to regular expressions. * Per #3148, update all calls to get_filenames() to pass regular expressions as arguments rather then file prefix and suffix strings. * Per #3148, use more consistent logging levels. * Per #3148, update ascii2nc unit tests to demonstrate providing inputs with file lists or searching input directories. * Per #3148, update the ascii2nc documentaiton. * Per #3148, refine the ascii2nc usage statement, refine log messages, and make sure python inputs work. * Per #3148, update ascii2nc to support multiple python inputs all specified as quoted strings on the command line, rather than only one. * Per #3148, correct the usage statement in the User's Guide. * Per #3148, fix typo in ascii2nc unit test. * Per #3148, define atcf_suffix and atcf_reg_exp in one spot in the vx_tc_util library instead of redefining it in each TC application. * Per #3148, address easy SonarQube code smells.g * Per #3148, correct logic in get_input_files()... had an if that should be else if. * Per #3148, whitespace diffs. * Per #3148, update the ascii2nc error-handling logic. Rather than a bad or missing filename causing ascii2nc to error out completely, process as many input files as it can and print warning messages about problematic inputs. * Not released to #3148, but coordinated with Howard Soh to update the formatting of the method_name strings that appear in log messages for consistency. * Per #3148, remove unneeded exclamation points * Per #3148, for file handlers, generally, change Error log messages to Warning since the top level file handler no longer exits with bad status when failing to read an input file. Instead, a warning is printed since processing will continue to the next file. * Per #3148, add DEBUG(2) log message to list the number of successful input file reads. * Per #3148, update ascii2nc to print the -format option being applied when the user has not provided one on the command line. --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
…l workflow for the develop branch.
* Unrelated to #3206, increase the METbaseimage version number from v3.4 to v3.4.1 so that the resulting Docker images contain 0 critical vulnerabilities. * Per #3206, update met_regrid_area_weighted() function to fix bug when OMP_NUM_THREADS is set very large. Need to use a reduction for vectors of doubles to prevent two threads from updating the same to_data value at the same time. * Per #3206, add comment.
* Fix jobstring to write -out_wind_logic correctly instead of reporting the value for cnt_logic instead. * Restore logic in aggr_mpr_wind_lines() removed for #2395 when STAT-Analysis aggregates MPR lines to compute WDIR output. In main_v11.0, that job automatically removes 0 vectors when computing ROW_MEAN_WDIR output. * Per #3210, update the revision history * Per #3210, update the WDIR documentation * Per #3210, add one more WDIR job to STAT-Analysis using all inputs, not just the non-zero MPR vectors. * Unrelated to #3210, but noted by @CPKalb, note that MET can read raw WRF output. * Per #3210, solve one additional bug in the MPR -> VCNT aggregation. Since the local v_info object was declared outside the loop, when d_diff is bad data, the values from the previous iteration were being using. This cause the TOTAL_DIR value to be reported as the TOTAL value and not properly skip over the zero length vectors as it should... and as Point-Stat does. --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
* #3172 Added get_grib_var_name * #3172 Added grib_code_map * #3172 Fixed SonarQube findings * #3172 Use the grib variable name instead of grib_code from the configuration * #3172 Added grib_unit_map * 172 Added var_units and read it * 172 Added var_units_set and process_var_units * 172 Calls process_var_units if use_var_id is true * #3172 Added parse_conf_key_values_map * #3172 Use an integer key for grib code * #3172 Added point_stat_QTY_INC_EXC_MADIS * Defined grib_var_map * #3172 Cleanup for SonarQube findings * #3172 Cleanup for SonarQube findings * #3172 FIx SonarQube findings * #3172 Added var_descs * #3172 Added grib_var_map for madis2nc * #3172 Fix SonarQube findings * #3172 CHeck var_names and var_units at process_var_units * #3173 Modified the comment for grib_var_map * #3172 Rollback (removed point_stat_QTY_INC_EXC_MADIS) --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
* Per #3219, add GlobalPython:set_args() function to store the Python path prior to calling Py_InitializeFromConfig() and then restore the Python path afterwards. For Python 3.10, the call to Py_InitializeFromConfig() resets the Python path while in Python 3.12 it does not. * Per #3219, update logic to eliminate duplicated log messages. * Per #3219, working toward consistency in python embedding logic, function names, and log messages. * Per #3219, rename wrappers_dir as pyembed_dir and make its location correction. I'm not positive we need it in the python path, but figure its safer to correct its location than it is to remove it. * Per #3219, don't need ascii2nc to add pyembed_dir to the path since GP.initialize() already adds it. * Per #3219, make logic, function names, and logging consistent in python_pointdata.cc * Per #3219, more consistency in function names and log messages. * work in progress * Per #3219, fix met_python_point_data() Python import command. * Per #3219, fix problem with preparing arguments for TC-Diag Python embedding. * Per #3219, add note to the TC-Diag chapter about the role of SciPy in computing diagnostics.
* Per #3168, update RMW-Analysis config files and corresponding documentation. * Per #3168, add ATCFLineBase::warning_hour() accessor function. * Per #3168, update DataCube to check for and handle bad data. * Per #3168, delete the unused track_point_dim. * Per #3168, commit working version of RMW-Analysis before eliminating dynamically allocated memory. * Per #3168, stop allocating/deallocting DataCub objects. And print warning when 0 track points are retained. * Per #3168, complete bad data checks in data_cube.cc. * Per #3168, update TC-Pairs to call ATCFLineBase::warning_hour() rather than computing it itself. * Per #3168, move parse_conf_thresh_map and parse_conf_string_map from TC-Stat into the vx_config library. * Per #3186, saving working state after adding the parsing for filtering options. Still need to update rmw_analysis.cc to actually apply those filtering options. * Not directly related to #3168, but update the TC-Pairs config file parsing to make better use of common library code. * Per #3168, update the RMW-Analysis config file parsing to make better use of common library code. * Not directly related to #3168, but update the TC-RMW config file parsing to make better use of common library code. * Per #3168, switch from Lead to LeadTime to match conventions in TC-RMW. * Per #3168, correct the URL to documentation of the ATCF file format. * Per #3168, move check_masks() from TC-Pairs and TC-Stat into the vx_statistics library. * Per #3168, add TrackPoint::get_atcf_val(string) member function. * Per #3168, update RMW-Analysis to validate the ATCF column names used. * Per #3168, update the RMWAnalysis config file and documentation. * Per #3168, tweak docs. * Per #3168, whitespace. * Per #3168, fix easy SonarQube code smells. * Per #3168, add Debug(3) log messages explaining why entire tracks and individual track points are being discarded. Also add a second RMW-Analysis unit test showing these filters in action. * Per #3168, convert seconds to an HHMMSS timestring for log messages. * Per #3168, update description of init_mask and valid_mask --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These differences are expected based on #3185.
ERROR: Differences were found with 2 files
NetCDF diff
A:MET-develop-ref/test_output/rmw_analysis/rmw_analysis_out.nc
B:MET-develop/test_output/rmw_analysis/rmw_analysis_out.nc
file not found (new output)
A:
B:MET-develop/test_output/rmw_analysis/rmw_analysis_out_filters.nc
Created by @JohnHalleyGotway