Fix Ensemble-Stat to work with different missing members for two or more variables #2208
Description
Replace italics below with details for this issue.
Describe the Problem
Provide a clear and concise description of the bug here.
This is reported by Binbin See discussion #1700
The ensemble_stat works with the ensemble input files with/without PRMSL (sea level pressure) and obs input data with PRMSL. If the other variable with different missing members (for example, VIS) is added, ensemble_stat fails.
- two missing members for PRMSL
- no missing members for VIS
export METPLUS_FCST_FIELD="field = [{ name=\"VIS\"; level=\"L0\"; },{ name=\"PRMSL\"; level=\"L0\"; }];";
export METPLUS_OBS_FIELD="field = [{ name=\"VIS\"; level=\"L0\"; },{ name=\"PRMSL\"; level=\"L0\"; }];";
export METPLUS_OBS_FIELD="field = [{ name=\"VIS\"; level=\"L0\"; },{ name=\"PRMSL\"; level=\"L0\"; }];";
Error message:
ERROR :
ERROR : NumArray::operator[] (int) const -> range check error ... Length = 0, i = 0
ERROR :
ensemble input files:
href.m01.t06z.conus.f06
href.m02.t06z.conus.f06
href.m03.t06z.conus.f06 - no PRMSL
href.m04.t06z.conus.f06 - no PRMSL
href.m05.t06z.conus.f06
href.m06.t06z.conus.f06
href.m07.t06z.conus.f06
href.m08.t06z.conus.f06
href.m09.t06z.conus.f06
href.m10.t06z.conus.f06
Command to reproduce the problem: This was prepared by @georgemccabe (building testing environments: copying input files, and setup the configurations) or run /d1/personal/hsoh/data/discussions_1700/run.sh
export METPLUS_CENSOR_THRESH="";
export METPLUS_CENSOR_VAL="";
export METPLUS_CI_ALPHA="ci_alpha = [0.05];";
export METPLUS_CLIMO_CDF_DICT="";
export METPLUS_CLIMO_MEAN_DICT="";
export METPLUS_CLIMO_STDEV_DICT="";
export METPLUS_CONTROL_ID="";
export METPLUS_DESC="desc = \"NA\";";
export METPLUS_DUPLICATE_FLAG="duplicate_flag = NONE;";
export METPLUS_ECLV_POINTS="";
export METPLUS_ENSEMBLE_FLAG_DICT="ensemble_flag = {latlon = FALSE;mean = FALSE;stdev = FALSE;minus = FALSE;plus = FALSE;min = FALSE;max = FALSE;range = FALSE;vld_count = FALSE;frequency = TRUE;nep = FALSE;nmep = FALSE;rank = FALSE;weight = FALSE;}";
export METPLUS_ENS_FIELD="field = [{ name=\"VIS\"; level=\"L0\"; },{ name=\"PRMSL\"; level=\"L0\"; }];";
export METPLUS_ENS_FILE_TYPE="";
export METPLUS_ENS_MEMBER_IDS="";
export METPLUS_ENS_OBS_THRESH="";
export METPLUS_ENS_PHIST_BIN_SIZE="";
export METPLUS_ENS_SSVAR_BIN_SIZE="";
export METPLUS_ENS_THRESH="ens_thresh = 0.4;";
export METPLUS_ENS_VLD_THRESH="vld_thresh = 0.4;";
export METPLUS_FCST_FIELD="field = [{ name=\"VIS\"; level=\"L0\"; },{ name=\"PRMSL\"; level=\"L0\"; }];";
export METPLUS_FCST_FILE_TYPE="";
export METPLUS_GRID_WEIGHT_FLAG="";
export METPLUS_INTERP_DICT="interp = {vld_thresh = 1.0;shape = SQUARE;type = {method = NEAREST;width = 1;}}";
export METPLUS_MASK_GRID="grid = [];";
export METPLUS_MASK_POLY="poly = [\"/d1/projects/METplus/METplus_Data/development/discussions/1700/g2o_href_prmsl/masks/grid2obs_CONUS.nc\"];";
export METPLUS_MESSAGE_TYPE="message_type = [\"ADPSFC\", \"SFCSHP\", \"MSONET\", \"ADPUPA\"];";
export METPLUS_MET_CONFIG_OVERRIDES="";
export METPLUS_MODEL="model = \"HREF\";";
export METPLUS_NBRHD_PROB_DICT="nbrhd_prob = {width = [3];shape = CIRCLE;vld_thresh = 0.0;}";
export METPLUS_NMEP_SMOOTH_DICT="nmep_smooth = {vld_thresh = 0.0;shape = CIRCLE;gaussian_dx = 81.27;gaussian_radius = 120;type = [{method = GAUSSIAN;width = 1;}];}";
export METPLUS_OBS_ERROR_FLAG="flag = FALSE;";
export METPLUS_OBS_FIELD="field = [{ name=\"VIS\"; level=\"L0\"; },{ name=\"PRMSL\"; level=\"L0\"; }];";
export METPLUS_OBS_FILE_TYPE="";
export METPLUS_OBS_QUALITY_EXC="";
export METPLUS_OBS_QUALITY_INC="";
export METPLUS_OBS_WINDOW_DICT="obs_window = {beg = -3600;end = 3600;}";
export METPLUS_OBTYPE="obtype = \"PREPBUFR\";";
export METPLUS_OUTPUT_FLAG_DICT="output_flag = {ecnt = BOTH;rps = NONE;rhist = BOTH;phist = NONE;orank = NONE;ssvar = NONE;relp = NONE;}";
export METPLUS_OUTPUT_PREFIX="output_prefix = \"HREF_PREPBUFR_CONUS_SFC_FHR06\";";
export METPLUS_PROB_CAT_THRESH="";
export METPLUS_PROB_PCT_THRESH="";
export METPLUS_REGRID_DICT="regrid = {to_grid = NONE;method = NEAREST;width = 1;vld_thresh = 0.5;shape = SQUARE;}";
export METPLUS_SKIP_CONST="skip_const = FALSE;";
export MET_OBS_ERROR_TABLE="";
export MET_TMP_DIR="/dev/shm/$LOGNAME/met_tmp";
OUTPUT_DIR="/dev/shm/$LOGNAME/met_test/stat/HREF/"
[ ! -d $OUTPUT_DIR ] && mkdir -p $OUTPUT_DIR
[ ! -d $MET_TMP_DIR ] && mkdir -p $MET_TMP_DIR
/d1/projects/MET/MET_releases/met-10.1.1/bin/ensemble_stat -v 10 \
/d1/personal/mccabe/out-b/stage/file_lists/20220718060000_6_ensemble_stat.txt \
/var/autofs/mnt/linux-amd64/debian/buster/local/METplus-4.1.0/parm/met_config/EnsembleStatConfig_wrapped \
-point_obs /d1/projects/METplus/METplus_Data/development/discussions/1700/g2o_href_prmsl/input_href.20220718/prepbufr.t12z.G227.nc \
-outdir $OUTPUT_DIR
Expected Behavior
Provide a clear and concise description of what you expected to happen here.
ensemble_stat should process the missing values regardless the order of the variables
Environment
Describe your runtime environment:
1. Machine: (e.g. HPC name, Linux Workstation, Mac Laptop) Reported at Hara and re-produce at seneca
2. OS: (e.g. RedHat Linux, MacOS) RedHat Linux
3. Software version number(s) MET 10.1.1
To Reproduce
Describe the steps to reproduce the behavior:
1. run above commands or /d1/personal/hsoh/data/discussions_1700/run.sh
2. See error
Post relevant sample data following these instructions:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-help-desk#ftp
Relevant Deadlines
List relevant project deadlines here or state NONE.
Funding Source
2792541
Define the Metadata
Assignee
- Select engineer(s) or no engineer required
- Select scientist(s) or no scientist required
Labels
- Select component(s)
- Select priority
- Select requestor(s)
Projects and Milestone
- Select Organization level Project for support of the current coordinated release
- Select Repository level Project for development toward the next official release or add alert: NEED PROJECT ASSIGNMENT label
- Select Milestone as the next bugfix version
Define Related Issue(s)
Consider the impact to the other METplus components.
Bugfix Checklist
See the METplus Workflow for details.
- Complete the issue definition above, including the Time Estimate and Funding Source.
- Fork this repository or create a branch of main_<Version>.
Branch name:bugfix_<Issue Number>_main_<Version>_<Description> - Fix the bug and test your changes.
- Add/update log messages for easier debugging.
- Add/update unit tests.
- Add/update documentation.
- Push local changes to GitHub.
- Submit a pull request to merge into main_<Version>.
Pull request:bugfix <Issue Number> main_<Version> <Description> - Define the pull request metadata, as permissions allow.
Select: Reviewer(s) and Linked issues
Select: Organization level software support Project for the current coordinated release
Select: Milestone as the next bugfix version - Iterate until the reviewer(s) accept and merge your changes.
- Delete your fork or branch.
- Complete the steps above to fix the bug on the develop branch.
Branch name:bugfix_<Issue Number>_develop_<Description>
Pull request:bugfix <Issue Number> develop <Description>
Select: Reviewer(s) and Linked issues
Select: Repository level development cycle Project for the next official release
Select: Milestone as the next official version - Close this issue.