Lastly, is there any way to write simple tests for this?

@dakinggg Since it's not really usable on current hardware I can add some negative tests where we check for the errors on current hardware. Something like this useful?

Is there a target for transformer_engine in setup.py

Also, is this something you want to add to setup.py as an optional dependency, or too soon?

Added a target txengine but two issues: 1) there is no pypi package for it 2) Installation depends on torch so it fails due to pip build isolation :-(

Yeah, a negative test that errors would be helpful for now. Just to exercise the code path. Thanks! And hm, installation depending on torch is unfortunate. In that case, could you put installation instructions somewhere? Probably in the documentation for it fp8?

@dakinggg added tests
@mvpatel2000 added installation instructions

New to fp8 or bf8. Does this mean it's possible to do fp8 training with consumer level ada GPU like rtx 4090?

@lukaemon : Two variants of FP8 exist and there is nothing called bf8 on NVidia cards.

training with consumer level ada GPU

I think so. Provided you have CUDA 12, TransformerEngine layers in your model and using amp_fp8 precision.
NVidia announcement does point to FP8 in 4090.

Ada’s new 4th Generation Tensor Cores are unbelievably fast, with an all new 8-Bit Floating Point (FP8) Tensor Engine, increasing throughput by up to 5X, to 1.32 Tensor-petaFLOPS on the GeForce RTX 4090.

@dskhudia Thanks for clarification.

@lukaemon @dskhudia No!!!

NVIDIA/TransformerEngine#15

@vgoklani Thanks. Ada fp8 support after 23q2 at least.

TransformerEngine has support for Ada GPUs now. Can the restriction in this PR be loosened to torch.cuda.get_device_capability() >= (8, 9)?

I've tested fp8 on a 4090 with llm-foundry and training starts successfully.