Skip to content

filter_valid_label: force scores tensor onto the selected device #502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Apr 7, 2022
Merged

filter_valid_label: force scores tensor onto the selected device #502

merged 1 commit into from
Apr 7, 2022

Conversation

ntw-au
Copy link
Contributor

@ntw-au ntw-au commented Mar 29, 2022 •
edited by germanros1987
Loading

This avoids a crash when calculating semantic segmentation loss during SemanticSegmentation.run_inference caused by the valid_scores and valid_labels tensors existing on separate devices.

This situation only arises when device is set to an accelerator, which makes valid_scores located on cpu and valid_labels located on cuda:0.

INFO - 2022-03-24 14:48:07,015 - train_evaluate - Launching inference for test[0]
Traceback (most recent call last):
  File "train_evaluate.py", line 157, in <module>
    result = pipeline.run_inference(test_instance)
  File "/opt/conda/lib/python3.6/site-packages/open3d/_ml3d/torch/pipelines/semantic_segmentation.py", line 175, in run_inference
    model.cfg.ignored_label_inds, device)
  File "/opt/conda/lib/python3.6/site-packages/open3d/_ml3d/torch/modules/losses/semseg_loss.py", line 20, in filter_valid_label
    valid_idx.unsqueeze(-1).expand(-1, num_classes))
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

This change is Reviewable

@sanskar107 sanskar107 self-requested a review April 7, 2022 14:34
@sanskar107 sanskar107 merged commit c8b8e1b into isl-org:dev Apr 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sanskar107 sanskar107 sanskar107 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ntw-au @sanskar107