Current implementation of PPO uses fixed learning rate for duration of training process. This can produce degenerate models later in training, when a smaller learning rate is necessary.

Learning rate should be annealed over time to 0.