Kernel type summaries:
  name                                       time   frac
  cutlass::Kernel                          161.29  75.94%
  softmax_autoregressive_backward_kernel     7.81   3.68%
  adamw_kernel2                              7.07   3.33%
  softmax_forward_kernel5                    5.61   2.64%
  fused_classifier_kernel3                   5.53   2.60%
  matmul_backward_bias_kernel4               5.07   2.39%
  layernorm_backward_kernel2                 4.55   2.14%
  gelu_backward_kernel                       3.55   1.67%
  gelu_forward_kernel                        2.40   1.13%
  layernorm_forward_kernel3                  2.25   1.06%
  permute_kernel                             1.92   0.90%
  permute_kernel_backward                    1.90   0.89%
  residual_forward_kernel                    1.85   0.87%
  unpermute_kernel_backward                  0.72   0.34%
  unpermute_kernel                           0.67   0.31%
  encoder_backward_kernel                    0.14   0.06%
  encoder_forward_kernel2                    0.08   0.04%

In total, a training step takes 212.4ms, distributed as:
  0.2ms (0.1%) in the encoder,
  54.8ms (25.8%) in forward blocks,
  48.9ms (23.0%) in the classifier part,
  101.4ms (47.8%) in backward blocks, and
  7.1ms (3.3%) in the optimizer.

We read 44.1GiB (207.8GB/s) and write 19.9GiB (93.5GB/s) to DRAM,
read 115.7GiB (544.6GB/s) and write 16.9GiB (79.4GB/s) to L2,
and execute 3.7 billion instructions (17.2 GInst/s).

python train_gpt2.py --write_tensors 0 --compile 1 --tensorcores 1 --num_iterations 100 --sequence_length 1024