I can probably share my experiences with Google Colab's T4 so far. I can say that it's not very feasible to train a high fidelity model of any kind that way.

The problem is VRAM of that single T4 - it's VERY small. I had to bring batch_size all the way down to 3 and max_len to 200. That means the model can only be properly trained on audio files with a maximum of 2,5 second in length. Any other combination (even batch_size = 2 and max_len = 200) would put the T4 out of memory during the first few epochs.

While that might be enough to train the model to some extent (with lots of epochs), your model probably wouldn't catch all the emotional nuances it needs to perform well on longer sentences and their context.

It could be possible to split audio samples into 2,5s clips where 2-3 words are spoken together with a certain emotion and train the model that way. However, that would be very time consuming - both the preparation phase and the training phase.

If anyone has more insights on how to optimize this more, so we could train our own StyleTTS2 from scratch on a T4, please don't hesitate to share them here.

You can rent a A6000 at vast ai for like .20-.30 cents an hour. It's super cheap. My data set was kinda big so it took awhile. I bet you could try maybe a 20 minute data set on the a6000 and get done before you spend a dollar.

The example I showed was for single speaker, but you can surely finetune on a multi-speaker dataset and it would work in the same way. You do not need to change anything, just prepare the dataset in the same way as the LibriTTS dataset.

I just got done with my last fine tuning run a few hours ago and I was playing with some of the parameters (diffusion, alpha, beta, etc) and I need to keep testing things out, but it sounds very good so far. I am in the process of using it as a base model for an RVC model to be mapped on top of. I have done this with Coqui XTTSv2 and the results were the best i've ever heard. It cleans up a lot of the issues when you have two models of the same speaker being used for inference.

Hi, firstly, want to thank for sharing all these information! Could you bit elaborate on the RVC model? Which model are you using? Is there any open source material I can read?

I think what he did is to train both a styletts2 and a rvc model on the same speaker dataset. Then using the styletts2 mode as the initial audio (to get the style, expressions ...) and then cleaning up the voice itself with the RVC.

Great! Please, if you see anything that is being done incorrectly or if there is a better option let me know.