Feature #2425 tc_stat_rirw #2536

sethlinden · 2023-05-05T20:02:37Z

Expected Differences

Do these changes introduce new tools, command line arguments, or configuration file options? [Yes]

If yes, please describe:

The -out_stat option can now be invoked for RIRW jobs. A new .stat output file will be created containing the 24 Common STAT Header columns along with either CTC and/or CTS header columns.

Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]

If yes, please describe:

Pull Request Testing

Describe testing already performed for these changes:

Tested using the following command line:

./tc_stat -lookin /d1/projects/MET/MET_regression/develop/NB20230503/MET-develop/test_output/tc_pairs/alal2010.tcst -job rirw -rirw_window 12 -rirw_thresh '<=-15' -out_line_type CTC,CTS -out_stat ./out.stat

Also called the same command line with -out_line_type CTC,CTS,MPR to verify that gives an Error message about not supporting MPR for the out_stat output file.

Added new test to TCStatConfig_ALAL2010:

"-job rirw -rirw_window 12 -rirw_thresh <=-15 -out_line_type CTC,CTS -out_stat ${MET_TEST_OUTPUT}/tc_stat/ALAL2010_rirw.stat"

Ran the unit test: unit_tc_stat.xml by hand

Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

Run the updated unit test for: unit_tc_stat.xml
Check the new stat output file:

../MET/test_output/tc_stat/ALAL2010_rirw.stat

Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]

Yes, I updated the documentation in: docs/Users_Guide/tc-stat.rst, related to the RIRW job and being able to use the -out_stat option

Do these changes include sufficient testing updates? [Yes]

Yes, updated the config file TCStatConfig_ALAL2010: used in unit_tc_stat.xml

Will this PR result in changes to the test suite? [Yes or No]

If yes, describe the new output and/or changes to the existing output:

Yes, it creates a new output file called:

../MET/test_output/tc_stat/ALAL2010_rirw.stat

Please complete this pull request review by [Fill in date].

Pull Request Checklist

See the METplus Workflow for details.

Review the source issue metadata (required labels, projects, and milestone).
Complete the PR definition above.
Ensure the PR title matches the feature or bugfix branch name.
Define the PR metadata, as permissions allow.
Select: Reviewer(s)
Select: Organization level software support Project or Repository level development cycle Project
Select: Milestone as the version that will include these changes
After submitting the PR, select Development issue with the original issue number.
After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
Close the linked issue and delete your feature or bugfix branch from GitHub.

…supplied. In serialize(), fixed bug for -out_stat option string to use StatFile. In TCStatJobRIRW::do_job(), added code to call do_output(*StatOut). SL ci-skip-all

…kip-all

…t_row and prototype for setup_stat_file(). SL ci-skip-all

…_stat_output. In progress. SL ci-skip-all

…skip-all

…m-row, num-col. SL ci-skip-all

…W:do_job to produce out file and stat file appropriately. SL ci-skip-all

…the file-stream object. Now getting output in the out.stat file. SL ci-skip-all

…at_output. Modified RIRWMapData struct. SL ci-skip-all

…RIRW process_pair() to get the new vars: Init, Lead, Valid. Worked on filling in StatHeader object in do_stat_output. SL

…riptions from input file). SL

…L ci-skip-all

…of the shc (StatHdrColumns) object. SL ci-skip-all

…ob to keep track of unique description and headers.

…ut. The DESC column was totally missing!

…E, and VX_MASK columns.

…p couts. Updated do_stat_output for CTS. SL ci-skip-all

…ut_stat option, also added example to rirw jobs. SL

…d stating the number of header columns as being 24. Always want to avoid those sorts of specifics because if/when that number changes, we'll almost definitely forget to update it here.wq

…at_rirw

…ncy and using it in the RIRW job.

JohnHalleyGotway

I approve of these changes. @sethlinden, I would like to merge this PR myself. We have 3 PR's that will change the output. Once they're all reviewed, I'll merge them all and then update the truth dataset only once.

Note that I did make some edits on this branch.

Removed specifics about the number of STAT header columns that might change in the future.
Added StringArray::add_uniq(string) and used it in tc_stat_job.cc to reduce the number of LOC.
Minor whitespace edits for consistent indents.

sethlinden · 2023-05-08T19:17:15Z

@JohnHalleyGotway thanks for the edits and small changes...they all look good and reasonable to me.

Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: Seth Linden <linden@seneca.rap.ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> Co-authored-by: Seth Linden <linden@ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: davidalbo <dave@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> fix #2389 develop flowchart (#2392) Fix Python environment issue (#2407) fix definitions of G172 and G220 based on comments in NOAA-EMC/NCEPLIBS-w3emc#157. (#2406) fix #2380 develop override (#2382) fix #2408 develop empty config (#2410) fix #2390 develop compile zlib (#2404) fix #2412 develop climo (#2422) fix #2437 develop convert (#2439) fix for develop, for #2437, forgot one reference to the search_parent for a dictionary lookup. fix #2452 develop airnow (#2454) fix #2449 develop pdf (#2464) fix #2402 develop sonarqube (#2468) fix #2426 develop buoy (#2475) fix 2518 dtypes appf docs (#2519) fix 2531 compilation errors (#2533) fix #2531 compilation_errors_configure (#2535)

Seth Linden and others added 25 commits April 5, 2023 16:00

Adding code to write stats to output file if -out_stat job option is …

c0ae655

…supplied. In serialize(), fixed bug for -out_stat option string to use StatFile. In TCStatJobRIRW::do_job(), added code to call do_output(*StatOut). SL ci-skip-all

Per issue #2425, added do_stat_output to TCStatJobRIRW class. SL ci-s…

5434121

…kip-all

Per issue #2425, stubbing in new function do_stat_output. SL ci-skip-all

c30a0c7

Per issue #2425, added vars used by do_stat_output, like stat_at, sta…

9362846

…t_row and prototype for setup_stat_file(). SL ci-skip-all

Per issue #2425, added setup_stat_file() function. Also working on do…

557b855

…_stat_output. In progress. SL ci-skip-all

Per issue #2425, moved setup_stat_file to TCStatJobRIRW class. SL ci-…

5a7979a

…skip-all

Per issue #2425, made some changes to do_stat_output in regards to nu…

3320517

…m-row, num-col. SL ci-skip-all

Per issue #2425, in do_stat_output, changed some testing. SL ci-skip-all

13a7257

Per #2425, initialize stat_row = 0.

40b9119

Per #2425, work in progress

ef713ca

Merge branch 'develop' into feature_2425_tc_stat_rirw

43e032f

Per issue #2425, removed mpr from setup_stat_output, updated TCJobRIR…

d44e5f0

…W:do_job to produce out file and stat file appropriately. SL ci-skip-all

Per issue #2425, in do_stat_output, added line to write the table to …

4226b60

…the file-stream object. Now getting output in the out.stat file. SL ci-skip-all

Per issue #2425, working on populating StatHdrColumns object in do_st…

9466ce1

…at_output. Modified RIRWMapData struct. SL ci-skip-all

Per issue #2425, added Init to RIRWMapData struct, modified TCStatJob…

933701d

…RIRW process_pair() to get the new vars: Init, Lead, Valid. Worked on filling in StatHeader object in do_stat_output. SL

Per issue #2425, worked on code to fill the RIRWMap Desc object (desc…

e6caf33

…riptions from input file). SL

Per issue #2425, in do_stat_output, added code to for shc.set_mask. S…

3931639

…L ci-skip-all

Per issue #2425, in do_stat_output, added code to fill in more parts …

1f3030d

…of the shc (StatHdrColumns) object. SL ci-skip-all

Per #2425, add StringArray::add_uniq() function and use it the RIRW j…

10e865a

…ob to keep track of unique description and headers.

Per #2425, correct the numbering of the columns for the TC-Pairs outp…

a583a86

…ut. The DESC column was totally missing!

Per #2425, work in progress. Added logic to populate the MODEL, OBTYP…

57d1e71

…E, and VX_MASK columns.

Merge branch 'develop' into feature_2425_tc_stat_rirw

1b2e62b

Per issue #2425, added better debugging to setup_stat_file. Cleaned u…

16989b9

…p couts. Updated do_stat_output for CTS. SL ci-skip-all

Per issue #2425, added a job for RIRW with the -out_stat option. SL

03720e4

Per issue #2425, For RIRW jobs, added sentences describing the new -o…

9c9d052

…ut_stat option, also added example to rirw jobs. SL

sethlinden added this to the MET 11.1.0 milestone May 5, 2023

sethlinden requested a review from JohnHalleyGotway May 5, 2023 20:02

JohnHalleyGotway changed the title ~~Feature 2425 tc stat rirw~~ Feature #2425 tc_stat_rirw May 8, 2023

JohnHalleyGotway added 2 commits May 8, 2023 10:45

Per #2425, no real change. Just trimming whitespace.

bdfc305

Per #2425, reworded details about RIRW in the TC-Stat chapter to avoi…

fae84c0

…d stating the number of header columns as being 24. Always want to avoid those sorts of specifics because if/when that number changes, we'll almost definitely forget to update it here.wq

JohnHalleyGotway added 2 commits May 8, 2023 10:59

Merge remote-tracking branch 'origin/develop' into feature_2425_tc_st…

5c3c11b

…at_rirw

Per #2425, adding StringArray::add_uniq(string) function for consiste…

1c0ae56

…ncy and using it in the RIRW job.

JohnHalleyGotway approved these changes May 8, 2023

View reviewed changes

JohnHalleyGotway merged commit 66ef8a9 into develop May 10, 2023

JohnHalleyGotway mentioned this pull request May 11, 2023

Update develop-ref after #2524, #2536, and #2537 #2541

Merged

JohnHalleyGotway linked an issue May 11, 2023 that may be closed by this pull request

Enhance TC-Stat to write the RIRW job CTC/CTS output to a .stat output file #2425

Closed

20 tasks

JohnHalleyGotway deleted the feature_2425_tc_stat_rirw branch December 19, 2024 16:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature #2425 tc_stat_rirw #2536

Feature #2425 tc_stat_rirw #2536

Uh oh!

sethlinden commented May 5, 2023

Uh oh!

JohnHalleyGotway left a comment

Uh oh!

sethlinden commented May 8, 2023

Uh oh!

Uh oh!

Feature #2425 tc_stat_rirw #2536

Feature #2425 tc_stat_rirw #2536

Uh oh!

Conversation

sethlinden commented May 5, 2023

Expected Differences

Pull Request Testing

Pull Request Checklist

Uh oh!

JohnHalleyGotway left a comment

Choose a reason for hiding this comment

Uh oh!

sethlinden commented May 8, 2023

Uh oh!

Uh oh!