Skip to content

Enhance Series-Analysis to read its own output and incrementally update output statistics over time #1371

@JohnHalleyGotway

Description

@JohnHalleyGotway

Describe the New Feature

This is a feature that was requested by the UK Met Office via met-help:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95578

They would like to be able to create gridded statistics over a longer time period that they can hold their model output and analyses on disk. To enable this, we'd need to enhance Series-Analysis to read it own output to aggregate stats over a longer time period.

After discussing details with the MetOffice on July 24, 2024, we decided to handle this as described below:

  1. Update Series-Analysis to make it easier to configure to write "ALL" of the CTC, PCT, SL1L2, and SAL1L2 columns.
  2. Add support for a new (-aggr) command line option to provide output from a previous run of Series-Analysis.
  3. When the -aggr option is provided, read the previously generated counts and partial sums. Prior to computing the output statistics, aggregate the previously generated counts and partial sums with the newly generated ones. Compute the output statistics from those aggregated values.

Some details...

  • Recommend doing this directly in Series-Analysis rather than creating a new tool to do the work.
  • It should be pretty easy for Series-Analysis to figure out if series analysis is reading it's own input files. But we should make the options clear. Instead of "-fcst" and "-obs" or "-both" consider a new command line option to tell the tool that you're passing its in its own output.
  • It may be tedious for the user to make sure they've enabled all the write output variables to be aggregated later. Consider a shorthand way for them to do so to make that easier.

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

3 days?

Sub-Issues

Consider breaking the new feature down into sub-issues.
None needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Split between MetOffice (2799991) and NOAA (2792543) and 2783544

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Review projects and select relevant Repository and Organization ones
  • Select milestone

Define Related Issue(s)

Consider the impact to the other METplus components.

New Feature Checklist

See the METplus Workflow for details.

  • Complete the issue definition above.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.

Metadata

Metadata

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions