Describe the New Feature

This feature was requested via met-help (https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97454) and discussed on a telecon with NRL on 11/19/2020. The task is to add new filtering options to filter matched pairs based on differences between the fcst, obs, and climatology values. This approach provides a way to do some bulk quality control on the data. While you can already filter forecast and observation data separately, there is currently no way to filter by their differences directly in the STAT tools.

The Stat-Analysis tool can already do using the -column_thresh job command option. For example:
-job aggregate_stat -line_type MPR -out_line_type CNT -column_thresh ABS(FCST-OBS) lt2
This job would only use input MPR lines where the absolute value of FCST-OBS is less than 2. The task here is to support that logic in the STAT tools directly.

While this obviously applies to Point-Stat and Ensemble-Stat when processing point observations, it could also be useful when doing grid-to-grid comparisons (esp with satellite data) in Grid-Stat, MODE, Wavelet-Stat, MTD, and Series-Analysis. Attempt to add support for this in the library code and minimize the changes the application code.

When filtering data for this new reason in Point-Stat, be sure to add a corresponding rejection count entry to track and report the number of points discarded.

At a minimum, support filtering out matched pairs by...

• thresholding FCST-OBS and OBS-CLIMO_MEAN
• if using both climo mean and standard deviation, discard obs based on the climo distribution (e.g. obs >CDP99 || obs <CDP1)
• Other filtering options to support here?

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the new feature down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
• Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

New Feature Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s), Project(s), Milestone, and Linked issues
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.
• threshold obs - climo mean
• threshold obs relative to climo dist

Be sure to include this in the rejection reason counts.