-
Notifications
You must be signed in to change notification settings - Fork 756
Closed
Description
Describe the issue:
After upgrading to v1.6, we noticed this strange behavior, where the program hangs on a sharded BAM that holds only alt-contig mapping reads.
Setup
- Operating system: on GCE via Google Life Sciences API (through Cromwell)
- DeepVariant version: v1.6
- Installation method (Docker, built from source, etc.): official v1.6 docker
- Type of data: Both PacBio HiFi and ONT (10.4), on GRCh38.
Steps to reproduce:
- Command
/opt/deepvariant/bin/run_deepvariant \
--model_type=PACBIO \
--ref=GCA_000001405.15_GRCh38_no_alt_analysis_set.fa \
--haploid_contigs chrX,chrY \
--par_regions_bed GRCh38.PAR.bed \
--reads=/cromwell_root/<sample_id>.alts.bam \
--output_vcf=/cromwell_root/dv_output/<sample_id>.alts.deepvariant.vcf.gz \
--output_gvcf=/cromwell_root/dv_output/<sample_id>.alts.deepvariant.g.vcf.gz \
--num_shards=16
- Relevant log
(note it says "0 examples", so I suspect it is when no examples are available, not just when there's only alt-mapping reads, e.g. if one simulates reads error-free from the reference itself, it probably will have the same issue)
/cromwell_root/tmp.cd83af44/tmpuzrx3yrs/make_examples.tfrecord-00011-of-00016.gz.example_info.json
I0203 17:23:03.253894 135328978921280 make_examples_core.py:2958] example_shape = None
I0203 17:23:03.254237 135328978921280 make_examples_core.py:2959] example_channels = [1, 2, 3, 4, 5, 6, 7, 9, 10]
I0203 17:23:03.255900 135328978921280 make_examples_core.py:301] Task 11/16: Found 0 candidate variants
I0203 17:23:03.256017 135328978921280 make_examples_core.py:301] Task 11/16: Created 0 examples
I0203 17:23:04.930985 137565708298048 make_examples_core.py:301] Task 7/16: Writing example info to /cromwell_root/tmp.cd83af44/tmpuzrx3yrs/make_examples.tfrecord-00007-of-00016.gz.example_info.json
I0203 17:23:04.931358 137565708298048 make_examples_core.py:2958] example_shape = None
I0203 17:23:04.931699 137565708298048 make_examples_core.py:2959] example_channels = [1, 2, 3, 4, 5, 6, 7, 9, 10]
I0203 17:23:04.933463 137565708298048 make_examples_core.py:301] Task 7/16: Found 0 candidate variants
I0203 17:23:04.933572 137565708298048 make_examples_core.py:301] Task 7/16: Created 0 examples
I0203 17:23:09.199501 136895166957376 make_examples_core.py:301] Task 13/16: Writing example info to /cromwell_root/tmp.cd83af44/tmpuzrx3yrs/make_examples.tfrecord-00013-of-00016.gz.example_info.json
I0203 17:23:09.199875 136895166957376 make_examples_core.py:2958] example_shape = None
I0203 17:23:09.200180 136895166957376 make_examples_core.py:2959] example_channels = [1, 2, 3, 4, 5, 6, 7, 9, 10]
I0203 17:23:09.201941 136895166957376 make_examples_core.py:301] Task 13/16: Found 0 candidate variants
I0203 17:23:09.202048 136895166957376 make_examples_core.py:301] Task 13/16: Created 0 examples
real 112m20.375s
user 1760m59.767s
sys 11m47.541s
***** Running the command:*****
time /opt/deepvariant/bin/call_variants --outfile "/cromwell_root/tmp.cd83af44/tmpuzrx3yrs/call_variants_output.tfrecord.gz" --examples "/cromwell_root/tmp.cd83af44/tmpuzrx3yrs/make_examples.tfrecord@16.gz" --checkpoint "/opt/models/pacbio"
/usr/local/lib/python3.8/dist-packages/tensorflow_addons/utils/tfa_eol_msg.py:23: UserWarning:
TensorFlow Addons (TFA) has ended development and introduction of new features.
TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024.
Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP).
For more information see: https://github.com/tensorflow/addons/issues/2807
warnings.warn(
I0203 17:23:14.218397 132068663560000 call_variants.py:471] Total 1 writing processes started.
W0203 17:23:14.224790 132068663560000 call_variants.py:482] Unable to read any records from /cromwell_root/tmp.cd83af44/tmpuzrx3yrs/make_examples.tfrecord@16.gz. Output will contain zero records.
I0203 17:23:14.225926 132068663560000 call_variants.py:623] Complete: call_variants.
And then the program hangs there for 10+ hours (UTC time when I'm reporting is Feb. 04, 04:05, and the program still appears running).
We've observed this for both ONT and HiFi data on multiple samples, further suggesting this isn't a data issue.
Thanks!
Steve
Metadata
Metadata
Assignees
Labels
No labels