Python embedding using several MET tools: implementation unclear from docs #1605

Wesley-J-Davis · 2022-05-06T17:33:59Z

Wesley-J-Davis
May 6, 2022

I am attempting to utilize either regrid_data_plane or shift_data_plane to convert my data to a netcdf format so that I can use the gen_vx_mask tool to divide the data I'm investigating into smaller segments so that I can use the mode tool on these smaller regions.

I was unable to use python embedding to use the gen_vx_mask tool outright, so this is a workaround for that issue.

Below is a template for using shift_data_plane to create a .nc file from a .grib file

`shift_data_plane \
nam.grib \
nam_shift_APCP_12.nc \
'name = "APCP"; level = "A12";' \
-from 38.6272  -90.1978 \
-to   40.1717 -105.1092 \
-v 2`

I tried to use python embedding to accomplish this:

`shift_data_plane \
./DATA_DIR/high_utc12.h5 \
VACAPES_NE.nc \
'name="/lib/coamps_reader.py \
MET_PYTHON_INPUT_ARG coamps_sfc_ci-mask-cth-co_utc12_tau0_predictor_2018010112-2019123112:2019123112";' \
-from 34.00 -84.00 \
-to 34.00 -84.00 \
-v 2`

No errors are returned, the command simply doesn't run and the following text is displayed:


*** Model Evaluation Tools (METV9.1) ***

Usage: shift_data_plane
        input_filename
        output_filename
        field_string
        -from lat lon
        -to   lat lon
        [-method type]
        [-width n]
        [-shape SHAPE]
        [-log file]
        [-v level]
        [-compress level]

        where   "input_filename" is the name of a gridded data file to be plotted (required).
                "output_filename" is the name of the output NetCDF file to be written (required).
                "field_string" defines the data to be shifted from the input file (required).
                "-from lat lon" specifies the starting latitude (degrees north) and longitude (degrees east) to define the shift.
                "-to lat lon" specifies the ending latitude (degrees north) and longitude (degrees east) to define the shift.
                "-method type" overrides the default interpolation method (DW_MEAN) (optional).
                "-width" overrides the default interpolation width (2) (optional).
                "-shape" overrides the default interpolation shape (SQUARE) (optional).
                "-log file" outputs log messages to the specified file (optional).
                "-v level" overrides the default level of logging (2) (optional).
                "-compress level" overrides the compression level of NetCDF variable (optional).

Answered by Wesley-J-Davis

May 9, 2022

shift_data_plane /p/work1/daviswj/UV-122/uv/nrl/veriftools/lib/coamps_reader.py VACAPES_NE.nc 'name="MET_PYTHON_INPUT_ARG"; file_type=PYTHON_NUMPY;' -from 34.00 -84.00 -to 34.00 -84.00 -v 2

This command ran successfully. I had to eliminate any arguments that would usually be passed into the python reader and just hard code it all in there.

View full answer

JohnHalleyGotway · 2022-05-06T21:49:36Z

JohnHalleyGotway
May 6, 2022
Maintainer

@Wesley-J-Davis reading through this section of Appendix F of the MET User's Guide, I think you're very close. That section lists two calls to plot_data_plane that are equivalent, one using the PYTHON_NUMPY keyword and one using MET_PYTHON_INPUT_ARG.

Here's the first one using PYTHON_NUMPY where you'd normally have an input file name:

plot_data_plane PYTHON_NUMPY fcst.ps \
  'name="scripts/python/read_ascii_numpy.py data/python/fcst.txt FCST";' \
  -title "Python enabled plot_data_plane"

And the second one using MET_PYTHON_INPUT_ARG:

plot_data_plane data/python/fcst.txt fcst.ps \
  'name="scripts/python/read_ascii_numpy.py MET_PYTHON_INPUT_ARG FCST"; \
   file_type=PYTHON_NUMPY;' \
  -title "Python enabled plot_data_plane"

I see that you're doing the latter, but the one important detail you're missing is setting file_type = PYTHON_NUMPY;. Hopefully adding that to your call will make it run.

In general when a MET tool returns bad status and prints the usage statement, that means there was some problem in parsing the required arguments. It's printing the usage statement to tell you to look closely at the args. But I'm not totally sure why that's happening in this case.

I don't fully understand why you're running shift_data_plane... but once we get past this issue, we could discuss that some more.

0 replies

Wesley-J-Davis · 2022-05-09T18:00:44Z

Wesley-J-Davis
May 9, 2022
Author

I'm running shift_data_plane simply to convert the python files to .nc files. I'm shifting zero degrees to just get the .nc file to work with so I can run gen_vx_mask without all the PYTHON_NUMPY troubleshooting

The old command:
*Bear in mind that I'm running this through a singularity container that is running MET v9.1
**That caused an issue in and of itself because I was using the singularity run command and not the singularity exec. that issue is fixed and now we have errors to help in troubleshooting

shift_data_plane \
./DATA_DIR/high_utc12.h5 \
VACAPES_NE.nc \
'name="/lib/coamps_reader.py \
MET_PYTHON_INPUT_ARG coamps_sfc_ci-mask-cth-co_utc12_tau0_predictor_2018010112-2019123112:2019123112";' \
-from 34.00 -84.00 \
-to 34.00 -84.00 \
-v 2

The error

FileNotFoundError: [Errno 2] No such file or directory: '/path/to/DATA_DIR/high_utc12.h5.py'

It was treating the path to the actual file as the path to the reader. So, I change up the command and the structure of the reader to match it

shift_data_plane \
/path/to/coamps_reader.py \
VACAPES_NE.nc \
'name="MET_PYTHON_INPUT_ARG coamps_sfc_ci-mask-cth-co_utc12_tau0_predictor_2018010112-2019123112:2019123112";\
file_type=PYTHON_NUMPY;' -from 34.00 -84.00 -to 34.00 -84.00 -v 2

The error:

File "/p/work1/daviswj/UV-122/uv/nrl/veriftools/lib/coamps_reader.py", line 10, in <module>
    print (sys.argv[1])
IndexError: list index out of range

Now, I expect this sys.argv[1] to be: coamps_sfc_ci-mask-cth-co_utc12_tau0_predictor_2018010112-2019123112:2019123112, because that's what prints out when I run the reader using the command:

$MET_PYTHON_EXE /path/to/coamps_reader.py coamps_sfc_ci-mask-cth-co_utc12_tau0_predictor_2018010112-2019123112:2019123112

So what I see happening here is that the fcst_file argument in shift_data_plane is being treated as if it's supposed to be a python script. So it feels like I should be putting the path to the reader in this place. But then when I do, and the program is called, the sys.argv[] logic no longer works because the actual command being run by the MET internals is:

/path/to/$MET_PYTHON_EXE /path/to/met-9.1_share/wrappers/write_pickle_dataplane.py /tmp/tmp_met_pickle_257614_0 /path/to/coamps_reader.py

Therefore, the arguments that I'm intending to pass in to the script are being truncated.

Stands to reason that perhaps I should pass the python reader arguments in the fcst_file argument of shift_data_plane.

The command:

shift_data_plane coamps_sfc_ci-mask-cth-co_utc12_tau0_predictor_2018010112-2019123112:2019123112 VACAPES_NE.nc 'name="/path/to/coamps_reader.py MET_PYTHON_INPUT_ARG"; file_type=PYTHON_NUMPY;' -from 34.00 -84.00 -to 34.00 -84.00 -v 2

The error:

FileNotFoundError: [Errno 2] No such file or directory: 'coamps_sfc_ci-mask-cth-co_utc12_tau0_predictor_2018010112-2019123112:2019123112.py'

Once again, treating the fcst_file argument as if it's supposed to be the file reader
The internal command:

/path/to/MET_PYTHON_EXE/python3.6 /path/to/met-9.1_share/wrappers/write_pickle_dataplane.py /tmp/tmp_met_pickle_250187_0 coamps_sfc_ci-mask-cth-co_utc12_tau0_predictor_2018010112-2019123112:2019123112

Next, I abandon the MET_PYTHON_INPUT_ARG method for the PYTHON_NUMPY method

The command:

shift_data_plane \
PYTHON_NUMPY \
VACAPES_NE.nc \
'name="/path/to/coamps_reader.py coamps_sfc_ci-mask-cth-co_utc12_tau0_predictor_2018010112-2019123112:2019123112";' \
-from 34.00 -84.00 -to 34.00 -84.00 -v 2

The error:

Traceback (most recent call last):
  File "/p/work1/daviswj/uv/nrl/veriftools/data/met-9.1_share/wrappers/write_pickle_dataplane.py", line 32, in <module>
    spec.loader.exec_module(met_in)
  File "<frozen importlib._bootstrap_external>", line 674, in exec_module
  File "<frozen importlib._bootstrap_external>", line 780, in get_code
  File "<frozen importlib._bootstrap_external>", line 832, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'PYTHON_NUMPY.py'

0 replies

Wesley-J-Davis · 2022-05-09T19:26:33Z

Wesley-J-Davis
May 9, 2022
Author

shift_data_plane /p/work1/daviswj/UV-122/uv/nrl/veriftools/lib/coamps_reader.py VACAPES_NE.nc 'name="MET_PYTHON_INPUT_ARG"; file_type=PYTHON_NUMPY;' -from 34.00 -84.00 -to 34.00 -84.00 -v 2

This command ran successfully. I had to eliminate any arguments that would usually be passed into the python reader and just hard code it all in there.

5 replies

JohnHalleyGotway May 12, 2022
Maintainer

@Wesley-J-Davis I don't think I fully appreciate all the details here. But from your perspective, is there something that should obviously be improved? Or is the python embedding logic actually working the ways it's documented... and the way you'd expect?

Wesley-J-Davis May 20, 2022
Author

First, sorry for the delay. I got caught up with work.

So, it feels like the documentation on the usage of the MET_PYTHON_INPUT_ARG was different than how it actually worked.

I expected to be able to do:

shift_data_plane argument_to_be_parsed_by_file_reader Obs.nc 'name="/path/to/file_reader.py MET_PYTHON_INPUT_ARG"; file_type=PYTHON_NUMPY;'

The MET_PYTHON_INPUT_ARG should be getting replaced with argument_to_be_parsed_by_file_reader in the pickle routine.

But that failed because the argument didn't make it into the pickle routine that integrates my python reader with met. So that means instead of being able to pass an argument with the MET_PYTHON_INPUT_ARG method and being able to control what gets pulled out of the file using an argument that I can change in my workflow logic (for instance, using say: CUSTOM_LOOP_LIST in METplus), I have to set environment variables that control the functions of the file reader using my own workflow that I have to build, which I don't think is the intended functionality of the MET_PYTHON_INPUT_ARG feature

JohnHalleyGotway Jun 7, 2022
Maintainer

@Wesley-J-Davis, I started by testing this out using the plot_data_plane command. The following 2 commands do produce the same result:

Use PYTHON_NUMPY on the command line where the gridded file name would typically go:

bin/plot_data_plane PYTHON_NUMPY plot.ps \
'name="scripts/python/read_ascii_numpy.py data/python/fcst.txt FCST";'

Instead, use the MET_PYTHON_INPUT_ARG option along with file_type=PYTHON_NUMPY;

bin/plot_data_plane data/python/fcst.txt plot.ps \
'name="scripts/python/read_ascii_numpy.py MET_PYTHON_INPUT_ARG FCST"; \
file_type=PYTHON_NUMPY;'

This is the intended functionality.

However, when I try to use these approaches with shift_data_plane, they DO NOT work.

With PYTHON_NUMPY:

bin/shift_data_plane PYTHON_NUMPY out.nc \
'name="scripts/python/read_ascii_numpy.py data/python/fcst.txt FCST";' \
-from 30 -110 -to 35 -115

Results in this error:

ERROR  : process_data_file() -> can't open file "PYTHON_NUMPY"

With MET_PYTHON_INPUT_ARG:

bin/shift_data_plane data/python/fcst.txt out.nc \
'name="scripts/python/read_ascii_numpy.py MET_PYTHON_INPUT_ARG FCST"; file_type=PYTHON_NUMPY;' \
-from 30 -110 -to 35 -115

Results in this error:

ERROR  : process_data_file() -> can't open file "data/python/fcst.txt"

This is a bug and we'll need to fix the shift_data_plane utility to correctly handle python embedding.

I'll moved this over to a new GitHub bugfix issue. Thanks for letting us know about the unexpected behavior!

JohnHalleyGotway Jun 7, 2022
Maintainer

@Wesley-J-Davis I propose that we mark this discussion as being "answered" and move any future conversation about it over to the dtcenter/MET#2173 issue. Any objections?

Wesley-J-Davis Jun 9, 2022
Author

none at all, and I really appreciate the help with this issue!

Python embedding using several MET tools: implementation unclear from docs #1605

Uh oh!

Wesley-J-Davis May 6, 2022

Replies: 3 comments · 5 replies

Uh oh!

Uh oh!

JohnHalleyGotway May 6, 2022 Maintainer

Uh oh!

Wesley-J-Davis May 9, 2022 Author

Uh oh!

Wesley-J-Davis May 9, 2022 Author

Uh oh!

JohnHalleyGotway May 12, 2022 Maintainer

Uh oh!

Wesley-J-Davis May 20, 2022 Author

Uh oh!

Uh oh!

JohnHalleyGotway Jun 7, 2022 Maintainer

Uh oh!

JohnHalleyGotway Jun 7, 2022 Maintainer

Uh oh!

Wesley-J-Davis Jun 9, 2022 Author

Wesley-J-Davis
May 6, 2022

Replies: 3 comments 5 replies

JohnHalleyGotway
May 6, 2022
Maintainer

Wesley-J-Davis
May 9, 2022
Author

Wesley-J-Davis
May 9, 2022
Author

JohnHalleyGotway May 12, 2022
Maintainer

Wesley-J-Davis May 20, 2022
Author

JohnHalleyGotway Jun 7, 2022
Maintainer

JohnHalleyGotway Jun 7, 2022
Maintainer

Wesley-J-Davis Jun 9, 2022
Author