Skip to content

W&B: fix dpp with wandb disabled #5163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Oct 13, 2021
Merged

W&B: fix dpp with wandb disabled #5163

merged 2 commits into from
Oct 13, 2021

Conversation

AyushExel
Copy link
Contributor

@AyushExel AyushExel commented Oct 13, 2021 •
edited by UltralyticsAssistant
Loading

Addresses #5160

🛠️ PR Summary

Made with ❤️ by Ultralytics Actions

🌟 Summary

Improved Weights & Biases (wandb) logging integration in YOLOv5.

📊 Key Changes

  • 📦 Added pkg_resources (as pkg) for version checking.
  • 🚀 Modified wandb login process with version check and a login attempt within a conditional block.
  • 💡 Moved wandb version check and login from wandb_utils.py to __init__.py within loggers.
  • 🌐 Ensured wandb login only occurs on the main process (when RANK is 0 or -1).

🎯 Purpose & Impact

  • 📈 Ensures better package version management for wandb, avoiding potential incompatibilities.
  • 🔒 Improves overall stability with a more controlled wandb login flow.
  • 🛠 Users will experience more reliable wandb integrations, especially in distributed training scenarios.
  • ⏱ Adds a timeout to wandb login for better failure handling.

@glenn-jocher glenn-jocher merged commit d0bfeb3 into ultralytics:master Oct 13, 2021
@glenn-jocher glenn-jocher mentioned this pull request Oct 13, 2021
Closed
@glenn-jocher glenn-jocher linked an issue Oct 13, 2021 that may be closed by this pull request
Docker Multi-GPU DDP training hang on destroy_process_group() with wandb option 3 #5160
Closed
@glenn-jocher
Copy link
Member

@AyushExel I think this PR might have introduced a new error. If I pull the latest image including this PR, run wandb login API_KEY, and then train normally, I get the following error:

Screen Shot 2021-10-13 at 6 37 59 PM

@Chigedaxigua
Copy link

@AyushExel I think this PR might have introduced a new error. If I pull the latest image including this PR, run wandb login API_KEY, and then train normally, I get the following error:

Screen Shot 2021-10-13 at 6 37 59 PM

I got the same issue today, did you slove this NameError yet?

@Chigedaxigua
Copy link

thank you!

@AyushExel AyushExel mentioned this pull request Oct 14, 2021
Merged
@AyushExel
Copy link
Contributor Author

@glenn-jocher ahh yes. wandb needs to be explicitly imported. Fix #5176

@glenn-jocher glenn-jocher mentioned this pull request Feb 22, 2022
Merged
BjarneKuehl pushed a commit to fhkiel-mlaip/yolov5 that referenced this pull request Aug 26, 2022
@AyushExel @glenn-jocher
W&B: fix DDP with wandb disabled (ultralytics#5163)
4ed96ec 
* fix dpp with wandb disabled

* PyCharm reformat

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Docker Multi-GPU DDP training hang on destroy_process_group() with wandb option 3
3 participants
@AyushExel @glenn-jocher @Chigedaxigua