Skip to content

how to get mixed precision to work (on 16gb *4080) #96

@rikabi89

Description

@rikabi89

(StyleTTS2) H:\StyleTTS2>accelerate launch --mixed_precision=fp16 train_first.py --config_path ./Configs/config_ft.yml The following values were not passed to accelerate launchand had defaults used instead:--num_processeswas set to a value of1 --num_machineswas set to a value of1 --dynamo_backendwas set to a value of'no'To avoid this warning pass in values for each of the problematic parameters or runaccelerate config`.
Traceback (most recent call last):
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\munch_init_.py", line 116, in getattr
return object.getattribute(self, k)
AttributeError: 'Munch' object has no attribute 'TMA_epoch'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\munch_init_.py", line 119, in getattr
return self[k]
KeyError: 'TMA_epoch'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train_first.py", line 444, in
main()
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\click\core.py", line 1157, in call
return self.main(*args, **kwargs)
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\click\core.py", line 783, in invoke
return _callback(*args, **kwargs)
File "train_first.py", line 131, in main
TMA_epoch = loss_params.TMA_epoch
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\munch_init
.py", line 121, in getattr
raise AttributeError(k)
AttributeError: TMA_epoch
Traceback (most recent call last):
File "H:\anaconda3\envs\StyleTTS2\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "H:\anaconda3\envs\StyleTTS2\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "H:\anaconda3\envs\StyleTTS2\Scripts\accelerate.exe_main
.py", line 7, in
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\accelerate\commands\accelerate_cli.py", line 45, in main
args.func(args)
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\accelerate\commands\launch.py", line 941, in launch_command
simple_launcher(args)
File "H:\anaconda3\envs\StyleTTS2\lib\site-packages\accelerate\commands\launch.py", line 603, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['H:\anaconda3\envs\StyleTTS2\python.exe', 'train_first.py', '--config_path', './Configs/config_ft.yml']' returned non-zero exit status 1.`

So it works without mixed precision but its extremely slow, in fact 1 epoch didnt even finish after 6 hours but this was a big dataset of around 2hours of wavs. Anything I can do optimise vram other than reduce batch and max_lens size? Win11 Anaconda

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions