Proposed change(s)

The Actor/Critic split will not allow us to evaluate the critic memories during inference (i.e, in policy.get_action(). This PR introduces a workaround, where the values are computed during get_trajectory_value_estimates() in the optimizer. We have to compute the value and resulting memories per sequence in each trajectory that is passed to this function.

Furthermore, initial memories per sequence were the current memories, not the previous (has been the case since forever). For instance, if you take a step at time t with obs_t, and get action_t and memory_t, when initializing the LSTM you want to use memory_(t-1). This PR fixes that for the policy memories.

Types of change(s)

• Bug fix
• New feature
• Code refactor
• Breaking change
• Documentation update
• Other (please describe)

Checklist

• Added tests that prove my fix is effective or that my feature works
• Updated the changelog (if applicable)
• Updated the documentation (if applicable)
• Updated the migration guide (if applicable)

Other comments