Refactored tracker health check code to become asynchronous. Adjusted a part of the tribler shutdown procedure as a result.

[lfdversluis]

More deletions than additions while keeping the same functionality (I hope), but much more clean!

• HTTP tracker code now makes use of Twisted's RedirectAgent. This means it will automatically redirect when a 30x code is returned, is asynchronous and has a timeout implemented.
• The UDP tracker is now asynchronous, has a timeout added and has some bug fixes.
• Shutdown of a UDP connection is now asynchronous, this is propogated in Tribler where it preparations are made for other components to shutdown asynchronous as well by using Deferreds.

Fixes #2035, contributing to #1848

Adds 29 tests
Improves responsiveness a lot, watchdog was tripping with > 10 seconds timeout. Not anymore.
Fixed 2 bugs
Improved redirection by making use of Twisted's redirectAgent
Added lots of documentation for other generations to read (enchancement)
Fixed 50+ pylint style errors

[lfdversluis]

Dirty reactor

[lfdversluis]

retest this please

[lfdversluis]

again dirty reactor

[lfdversluis]

retest this please

[lfdversluis]

Again a dirty reactor. I think the test gets a deferred now and terminates immediately. Will check tomorrow.

[lfdversluis]

retest this please

[lfdversluis]

It was correct! Let's see if 2/2 pass :)

[lfdversluis]

retest this please

[lfdversluis]

@whirm if this one succeeds too, I think it should be OK.

[lfdversluis]

retest this please

[whirm]

No need to retest, you need to fix this first:

ERROR   1458311111  test_as_server:TestGuiDialogs:122  The reactor was dirty:
ERROR   1458311111  test_as_server:TestGuiDialogs:124  >     <DelayedCall 0x7f1328ca0638 [48.7743420601s] called=0 cancelled=0 ThreadedResolver._cleanup('tracker.yify-torrents.com', <Deferred at 0x7f1328ca08c0>)>

[lfdversluis]

@whirm Dispersy is down according to @devos50. My code seems to be working fine but the tunnelcommunity tests are now failing and leaving stuff behind.

[whirm]

retest this please

[lfdversluis]

Jenkins is down, can't see the diff in coverage :/

[lfdversluis]

Looks like the hiccup is over

[lfdversluis]

@whirm Ok, tests are all passing on it again... I will squash now.

[whirm]

great!

[lfdversluis]

https://jenkins.tribler.org/job/GH_Tribler_PR_tests_linux/899/artifact/output/test_runners_output/00003.err/*view*/

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/jenkins/workspace/GH_Tribler_PR_tests_linux/tribler/Tribler/Core/Utilities/twisted_thread.py", line 55, in _reactor_runner
    reactor.run(installSignalHandlers=False)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1194, in run
    self.mainLoop()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1203, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 798, in runUntilCurrent
    f(*a, **kw)
  File "/home/jenkins/workspace/GH_Tribler_PR_tests_linux/tribler/Tribler/dispersy/endpoint.py", line 250, in dispersythread_data_came_in
    u"standalone_ep")
  File "/home/jenkins/workspace/GH_Tribler_PR_tests_linux/tribler/Tribler/dispersy/dispersy.py", line 1445, in on_incoming_packets
    assert all(len(packet[1]) > 22 for packet in packets), packets
exceptions.AssertionError: [(<Tribler.dispersy.candidate.Candidate object at 0x7f331c0a7ad0>, 'A\x00\xb7Y&&\x97\x01\x00\x00\x00\x00\x00\x10\x00\x00\x18\xc6\x00\x00')]

Haven't seen this one before.

[whirm]

I don't think receiving a 17byte datagram should raise an exception...

I'll have a look on the dispersy side.

[lfdversluis]

retest this please

[lfdversluis]

test_download_torrent_from_file (Tribler.Test.API.test_download.TestDownload) ... /home/jenkins/workspace/GH_Tribler_PR_tests_linux@2/gumby/scripts/wrap_in_temp_home.sh: line 34: 11081 Terminated

[lfdversluis]

retest this please

[lfdversluis]

Found an issue that may be related to my work

[lfdversluis]

@whirm tests passed 👍

[lfdversluis]

Huh again an one line but not the second covered issue. But this time not a continue

[whirm]

Looks fine, squash it

[lfdversluis]

retest this please

[whirm]

@lfdversluis are all tests passing locally for you? I find it a bit suspicious that the bucket that gets stuck is the one that contains the new tests.

[lfdversluis]

@whirm Yup Locally they run fine. Perhaps debian vs ubuntu? We know that download_from_url hangs on devel, and other tests

[whirm]

No sleeps please

[lfdversluis]

not my test unfortunately, just modified it

[whirm]

That sucks, it slipped right through.

@devos50 please, fix that test to not use sleep().

[devos50]

@whirm it's also not my test, I only made a modification to the test to let it use another torrent. I will create an issue.

[whirm]

To me it looks like the whole test is being added on this PR.

[whirm]

It's really weird.

[whirm]

Oh man, it's a Monday alright! That's a copy of the test over it... Those two tests are responsible for a full minute of test execution time. That's ridiculous.

[lfdversluis]

@whirm As you made a separate issue for the sleep(s), can this one be merged if all is/looks ok? AFAIK @ardhipoetra would like to rebase with this code as he has the current blocking code in his work now (and he depends on the statistics).

IF it turns out that there is indeed something weird going on with the tests (your gut feeling) besides the hanging stuff that will get fixed, I will open a new PR once I investigated.

[devos50]

I'm also dependent on this PR since I have to refactor the shutdown procedure of the HTTP API slightly as well so I would also like to see this merged before #2121.

[ardhipoetra]

Thanks for the merge! Will do the rebase