GAE helps a lot. It can give a advantage function with low varience and bias, much better than vinalla a3c.

pgql_online has the same performance as a3c in pong , just like the paper said.

pgql use replay memory without prioritized.