Enhance IODA2NC to support IODA v2.0 format #2068
Description
Describe the Enhancement
Rob from the Met Office received a sample IODA file from their DA group and the headers is very different from the currently supported format. Will M. took a look at it and confirmed the header is consistent with his understanding of the IODA v2.0 format. He is going to reach out to JEDI developers to determine if the format is set. If it is, then it would be good to add support for this format to IODA2NC. Here is the sample header that Rob sent. We will still need to get a sample file, but the file Rob had was very large and not conducive for testing.
Below is the header info which I got from running ncdump -h on the file I was given to look at: (Satwind data)
netcdf jopa_satwind_20210701T1200Z_out_0000 {
dimensions:
nlocs = UNLIMITED ; // (2815062 currently)
variables:
int nlocs(nlocs) ;
nlocs:suggested_chunk_dim = 10000LL ;
// global attributes:
string :_ioda_layout = "ObsGroup" ;
:_ioda_layout_version = 0 ;
group: DerivedObsError {
variables:
float eastward_wind(nlocs) ;
eastward_wind:_FillValue = -3.368795e+38f ;
float northward_wind(nlocs) ;
northward_wind:_FillValue = -3.368795e+38f ;
} // group DerivedObsError
group: DerivedObsValue {
variables:
float eastward_wind(nlocs) ;
eastward_wind:_FillValue = -3.368795e+38f ;
float northward_wind(nlocs) ;
northward_wind:_FillValue = -3.368795e+38f ;
} // group DerivedObsValue
group: Diag {
variables:
string satwind_id(nlocs) ;
string satwind_id:_FillValue = "*** MISSING ***" ;
} // group Diag
group: EffectiveError {
variables:
float eastward_wind(nlocs) ;
eastward_wind:_FillValue = -3.368795e+38f ;
float northward_wind(nlocs) ;
northward_wind:_FillValue = -3.368795e+38f ;
} // group EffectiveError
group: EffectiveQC {
variables:
int eastward_wind(nlocs) ;
eastward_wind:_FillValue = -2147483643 ;
int northward_wind(nlocs) ;
northward_wind:_FillValue = -2147483643 ;
} // group EffectiveQC
group: GrossErrorProbability {
variables:
float eastward_wind(nlocs) ;
eastward_wind:_FillValue = -3.368795e+38f ;
float northward_wind(nlocs) ;
northward_wind:_FillValue = -3.368795e+38f ;
} // group GrossErrorProbability
group: MetaData {
variables:
float PGE_u_over_v(nlocs) ;
PGE_u_over_v:_FillValue = -3.368795e+38f ;
float QI_common(nlocs) ;
QI_common:_FillValue = -3.368795e+38f ;
float QI_estimated_error(nlocs) ;
QI_estimated_error:_FillValue = -3.368795e+38f ;
float QI_recursive_filter_function(nlocs) ;
QI_recursive_filter_function:_FillValue = -3.368795e+38f ;
float QI_weighted_mixture_exc_forecast_comparison(nlocs) ;
QI_weighted_mixture_exc_forecast_comparison:_FillValue = -3.368795e+38f ;
float QI_with_forecast(nlocs) ;
QI_with_forecast:_FillValue = -3.368795e+38f ;
float QI_without_forecast(nlocs) ;
QI_without_forecast:_FillValue = -3.368795e+38f ;
float air_pressure(nlocs) ;
air_pressure:_FillValue = -3.368795e+38f ;
float air_pressure_original(nlocs) ;
air_pressure_original:_FillValue = -3.368795e+38f ;
int64 dateTime(nlocs) ;
dateTime:_FillValue = -9223372036854775801LL ;
string dateTime:units = "seconds since 1970-01-01T00:00:00Z" ;
float latitude(nlocs) ;
latitude:_FillValue = -3.368795e+38f ;
float longitude(nlocs) ;
longitude:_FillValue = -3.368795e+38f ;
int originating_centre(nlocs) ;
originating_centre:_FillValue = -2147483643 ;
int originating_subcentre(nlocs) ;
originating_subcentre:_FillValue = -2147483643 ;
float percent_confidence_1(nlocs) ;
percent_confidence_1:_FillValue = -3.368795e+38f ;
float percent_confidence_2(nlocs) ;
percent_confidence_2:_FillValue = -3.368795e+38f ;
float percent_confidence_3(nlocs) ;
percent_confidence_3:_FillValue = -3.368795e+38f ;
float percent_confidence_4(nlocs) ;
percent_confidence_4:_FillValue = -3.368795e+38f ;
int priority(nlocs) ;
priority:_FillValue = -2147483643 ;
float priority_numerator(nlocs) ;
priority_numerator:_FillValue = -3.368795e+38f ;
int64 receiptdateTime(nlocs) ;
receiptdateTime:_FillValue = -9223372036854775801LL ;
string receiptdateTime:units = "seconds since 1970-01-01T00:00:00Z" ;
int satellite_identifier(nlocs) ;
satellite_identifier:_FillValue = -2147483643 ;
string satwind_id(nlocs) ;
string satwind_id:_FillValue = "*** MISSING ***" ;
float sensor_central_frequency(nlocs) ;
sensor_central_frequency:_FillValue = -3.368795e+38f ;
float sensor_zenith_angle(nlocs) ;
sensor_zenith_angle:_FillValue = -3.368795e+38f ;
float total_observation_error(nlocs) ;
total_observation_error:_FillValue = -3.368795e+38f ;
int wind_computation_method(nlocs) ;
wind_computation_method:_FillValue = -2147483643 ;
int wind_generating_application_1(nlocs) ;
wind_generating_application_1:_FillValue = -2147483643 ;
int wind_generating_application_2(nlocs) ;
wind_generating_application_2:_FillValue = -2147483643 ;
int wind_generating_application_3(nlocs) ;
wind_generating_application_3:_FillValue = -2147483643 ;
int wind_generating_application_4(nlocs) ;
wind_generating_application_4:_FillValue = -2147483643 ;
} // group MetaData
group: MetaDataError {
variables:
float air_pressure(nlocs) ;
air_pressure:_FillValue = -3.368795e+38f ;
} // group MetaDataError
group: ObsBias {
variables:
float eastward_wind(nlocs) ;
eastward_wind:_FillValue = -3.368795e+38f ;
float northward_wind(nlocs) ;
northward_wind:_FillValue = -3.368795e+38f ;
} // group ObsBias
group: ObsValue {
variables:
float wind_from_direction(nlocs) ;
wind_from_direction:_FillValue = -3.368795e+38f ;
float wind_speed(nlocs) ;
wind_speed:_FillValue = -3.368795e+38f ;
} // group ObsValue
group: QCFlags {
variables:
int eastward_wind(nlocs) ;
eastward_wind:_FillValue = -2147483643 ;
int northward_wind(nlocs) ;
northward_wind:_FillValue = -2147483643 ;
} // group QCFlags
group: hofx {
variables:
float eastward_wind(nlocs) ;
eastward_wind:_FillValue = -3.368795e+38f ;
float northward_wind(nlocs) ;
northward_wind:_FillValue = -3.368795e+38f ;
} // group hofx
}
This issue is related to METplus issue #1453
Time Estimate
Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.
Sub-Issues
Consider breaking the enhancement down into sub-issues.
- Add a checkbox for each sub-issue here.
Relevant Deadlines
List relevant project deadlines here or state NONE.
Funding Source
2799991
Define the Metadata
Assignee
- Select engineer(s) or no engineer required
- Select scientist(s) or no scientist required
Labels
- Select component(s)
- Select priority
- Select requestor(s)
Projects and Milestone
- Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
- Select Milestone as the next official version or Future Versions
Define Related Issue(s)
Consider the impact to the other METplus components.
Enhancement Checklist
See the METplus Workflow for details.
- Complete the issue definition above, including the Time Estimate and Funding Source.
- Fork this repository or create a branch of develop.
Branch name:feature_<Issue Number>_<Description> - Complete the development and test your changes.
- Add/update log messages for easier debugging.
- Add/update unit tests.
- Add/update documentation.
- Push local changes to GitHub.
- Submit a pull request to merge into develop.
Pull request:feature <Issue Number> <Description> - Define the pull request metadata, as permissions allow.
Select: Reviewer(s) and Linked issues
Select: Repository level development cycle Project for the next official release
Select: Milestone as the next official version - Iterate until the reviewer(s) accept and merge your changes.
- Delete your fork or branch.
- Close this issue.