Possible problems in total_episode_reward_logger

[rasoolfa]

Hi,

1. Should here and here in total_episode_reward_logger be changed as belows:

                 for k in range(1, len(dones_idx[:, 0])):
                    rew_acc[env_idx] += sum(rewards[env_idx, dones_idx[k-1, 0]:dones_idx[k, 0]])
                    summary = tf.Summary(value=[tf.Summary.Value(tag="episode_reward", simple_value=rew_acc[env_idx])])
                    writer.add_summary(summary, steps + dones_idx[k, 0])
                rew_acc[env_idx] += sum(rewards[env_idx, dones_idx[-1, 0]:])

Notice the chanege for rew_acc[env_idx] = to rew_acc[env_idx] +=

2. steps + dones_idx[k, 0] (for all k) can be the same for different env_idx.
   Example:
   env_idx(1): steps + dones_idx(1000), simple_value = 20
   env_idx(7): steps + dones_idx(1000), simple_value = 120

In that case, will it be more appropriate to add average for those env_idxs with same steps + dones_idx rather than multiple entries?

[araffin]

Hello,

In that case, will it be more appropriate to add average for those env_idxs with same steps + dones_idx rather than multiple entries?

I think we need to improve that function anyway (see issues #224 and #143), and using average would be a solution. I would be glad if we could simplify that also, using Episode Stats maybe.