Terminating after "Plotting labels..." when training #5395
Description
While I am able to use YOLOv5 for inference, the train.py does not seem to work for me anymore. It did work previously however.
I have tried to clone the latest repo as well. I have set up a fresh Conda environment with Python 3.8. Again, inference works, but not training my custom data.
It will create the "exp" directory (exp24) in this case. Which contains an empty "weights" directory, hyp.yaml, opt.aml, and events.out.fs.events..0. No .pt, no images, no results.csv.
I have tried both the training set that I previously was able to train with and a new one I just created.
I run it using
python train.py --img 640 --batch 4 --epochs 200 --data C:/Users/kristofferk/Documents/GitHub/p9-api/experiment/kristoffer/step06-data.yaml --weights yolov5s.pt
But when it comes to "Plotting labels..." it will be stuck there for about 20 seconds and then terminate without any further warnings or errors.
The output of running train.py is:
PS C:\Users\kristofferk\Documents\GitHub\yolov5> python train.py --img 640 --batch 4 --epochs 200 --data C:/Users/kristofferk/Documents/GitHub/p9-api/experiment/kristoffer/step06-data.yaml --weights yolov5s.pt
wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
train: weights=yolov5s.pt, cfg=, data=C:/Users/kristofferk/Documents/GitHub/p9-api/experiment/kristoffer/step06-data.yaml, hyp=data\hyps\hyp.scratch.yaml, epochs=200, batch_size=4, imgsz=640, rect=False, resume=False, nosave=False, noval=False, noautoanchor=False, evolve=None, bucket=, cache=None, image_weights=False, device=, multi_scale=False, single_cls=False, adam=False, sync_bn=False, workers=8, project=runs\train, name=exp, exist_ok=False, quad=False, linear_lr=False, label_smoothing=0.0, patience=100, freeze=0, save_period=-1, local_rank=-1, entity=None, upload_dataset=False, bbox_interval=-1, artifact_alias=latest
github: up to date with https://github.com/KristofferK/yolov5
YOLOv5 v6.0-38-gc0c15d8 torch 1.8.2 CUDA:0 (NVIDIA GeForce RTX 3060, 12288.0MB)
hyperparameters: lr0=0.01, lrf=0.1, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=0.05, cls=0.5, cls_pw=1.0, obj=1.0, obj_pw=1.0, iou_t=0.2, anchor_t=4.0, fl_gamma=0.0, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0
TensorBoard: Start with 'tensorboard --logdir runs\train', view at http://localhost:6006/
wandb: Tracking run with wandb version 0.12.6
wandb: Syncing run stilted-monkey-10
wandb: View project at https://wandb.ai/kristofferk/train
wandb: View run at https://wandb.ai/kristofferk/train/runs/o18wqty1
wandb: Run data is saved locally in C:\Users\kristofferk\Documents\GitHub\yolov5\wandb\run-20211029_130316-o18wqty1
wandb: Run `wandb offline` to turn off syncing.
Overriding model.yaml nc=80 with nc=7
from n params module arguments
0 -1 1 3520 models.common.Focus [3, 32, 3]
1 -1 1 18560 models.common.Conv [32, 64, 3, 2]
2 -1 1 18816 models.common.C3 [64, 64, 1]
3 -1 1 73984 models.common.Conv [64, 128, 3, 2]
4 -1 3 156928 models.common.C3 [128, 128, 3]
5 -1 1 295424 models.common.Conv [128, 256, 3, 2]
6 -1 3 625152 models.common.C3 [256, 256, 3]
7 -1 1 1180672 models.common.Conv [256, 512, 3, 2]
8 -1 1 656896 models.common.SPP [512, 512, [5, 9, 13]]
9 -1 1 1182720 models.common.C3 [512, 512, 1, False]
10 -1 1 131584 models.common.Conv [512, 256, 1, 1]
11 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
12 [-1, 6] 1 0 models.common.Concat [1]
13 -1 1 361984 models.common.C3 [512, 256, 1, False]
14 -1 1 33024 models.common.Conv [256, 128, 1, 1]
15 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
16 [-1, 4] 1 0 models.common.Concat [1]
17 -1 1 90880 models.common.C3 [256, 128, 1, False]
18 -1 1 147712 models.common.Conv [128, 128, 3, 2]
19 [-1, 14] 1 0 models.common.Concat [1]
20 -1 1 296448 models.common.C3 [256, 256, 1, False]
21 -1 1 590336 models.common.Conv [256, 256, 3, 2]
22 [-1, 10] 1 0 models.common.Concat [1]
23 -1 1 1182720 models.common.C3 [512, 512, 1, False]
24 [17, 20, 23] 1 32364 models.yolo.Detect [7, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373,
326]], [128, 256, 512]]
Model Summary: 283 layers, 7079724 parameters, 7079724 gradients, 16.4 GFLOPs
Transferred 355/361 items from yolov5s.pt
Scaled weight_decay = 0.0005
optimizer: SGD with parameter groups 59 weight, 62 weight (no decay), 62 bias
train: Scanning 'C:\Users\kristofferk\Documents\GitHub\p9-api\experiment\kristoffer\datasets\malaria-yolov5\labels' images and labels...: 0%| | 0/50 [00:00<?, ?itwandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
train: Scanning 'C:\Users\kristofferk\Documents\GitHub\p9-api\experiment\kristoffer\datasets\malaria-yolov5\labels' images and labels...1 found, 0 missing, 0 empty,wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
train: Scanning 'C:\Users\kristofferk\Documents\GitHub\p9-api\experiment\kristoffer\datasets\malaria-yolov5\labels' images and labels...50 found, 0 missing, 0 empty
train: New cache created: C:\Users\kristofferk\Documents\GitHub\p9-api\experiment\kristoffer\datasets\malaria-yolov5\labels.cache
wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
val: Scanning 'C:\Users\kristofferk\Documents\GitHub\p9-api\experiment\kristoffer\datasets\malaria-yolov5\labels.cache' images and labels... 50 found, 0 missing, 0
wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
wandb: Currently logged in as: kristofferk (use `wandb login --relogin` to force relogin)
Plotting labels...
PS C:\Users\kristofferk\Documents\GitHub\yolov5>
Any suggestions on how to proceed from here? Either to fix it or at least get a more detailed error message.
Thanks in advance.