Connection: Implement a mechanism to retry failed HTTP requests for certain status_codes

[gzhytar]

Fixes #66
During peak times large amount of http requests are failing without a specific response from Kraken server. These are intermittent problems with CloudFlare/Kraken and for some of returned http codes it should be safe to retry an operation.
Max number of retries is configurable via bad_http_connection_retries attribute. As each request has a unique nonce which is checked on a server, it should prevent execution of several identical requests if they get to a server despite a failed status_code.

[veox]

On failure during peak use time, it is detrimental to resend the query without a "sleep" period.

Also, the fact that this doesn't create duplicate orders must be estabilished.

The default behaviour should be "no retries" IMO. It's also OK to call the object variable attribute the shorter self.retries, for less typing.

The status codes for which retries should be performed should be configurable.

Delaying until post-release of v2.0.0 (no changes to release candidate).

[veox]

I'm inclined to manually merge this into a feature-branch as-is, and then implementing the above notes; unless you have some commits not pushed to your github repo?..

[gzhytar]

@veox , hold on with this one. I've been testing it for a while and recently has got 2 cases where allegedly the same order was put twice (specifically when putting order i got a message "Insufficient funds", while I was able to see that order was successfuly created. I need to prove or refute the hypothesis that the message was generated by the second identical order as a result of query repeat) . Kraken was so unresponsive that time that it's not clear from logs whats exactly happened so i'm going to add some additional logging that will record NONCE passed with the query and let the group know back.

[veox]

The current load on Kraken is perfect for testing this code; I've gone ahead and worked it into branch retry-on-failure. (It's likely to see commit squashing, so I wouldn't recommend anyone basing anything off that.)

In general, it seems to work fine for me, at least ATM.

---

Regarding twin orders: do you have the "nonce window" setting changed? Did you acidentally add code 502 to those to retry on?

Anecdotally, right now a 502 is a 95%-certain indication that the query went through CloudFlare, and the trade execution engine was "free" enough to accept it (but not respond).

[veox]

Closing this in favour of the alternative approach in PR #78 - cleaned up in PR #99.

[veox]

Actually, opened a separate PR #100 that was based on this one, to compare approaches again...