ON HOLD: Tribler channel downloading experiment #282

ardhipoetra · 2016-11-08T15:29:37Z

This pull request contains a experiment working on the downloading activity via ChannelCommunity.

The steps are :

A node create ChannelCommunity, put this in the AllChannelCommunity
This node, create several .torrent files and put it into ChannelCommunity. At this point, this node already start seeding
Another node, try to find this channel via AllChannelCommunity. Then join it.
Another node download .torrent file via TFTP, and store it in the leveldbstore.
Start downloading.

This PR based on commit 89510a6 instead of HEAD because of issue #281.

Credit mining experiment will expand from this PR later on. See this and the comment above it.

tribler-ci · 2016-11-08T15:29:38Z

Can one of the admins verify this patch?

synctext · 2016-11-08T18:02:45Z

add to test

synctext · 2016-11-08T18:02:54Z

ok to test

synctext · 2016-11-08T18:03:06Z

add to whitelist

ardhipoetra · 2016-11-14T13:04:09Z

@devos50, I'm pretty much finished with this one. Do you want to review it? Or should we wait for the new gumby to be finished first?

devos50 · 2017-01-10T09:22:45Z

retest this please

devos50

I think it looks ok in general (only have a few comments), however, the commit history requires some attention. Please squash it to one or two commits. Also, don't forget to rebase.

After that, I will merge it.

devos50 · 2017-01-10T09:25:02Z

experiments/tribler/channel_download.py

+
+ count_out = 0
+ while self._dispersy is None and count_out < 100:
+ time.sleep(1.0)


Why is this sleep necessary?

Basically, it waits for dispersy to be properly started

Please use reactor.callLater instead, this will block the whole thing.

devos50 · 2017-01-10T09:25:40Z

experiments/tribler/channel_download.py

+
+ from Tribler.Core.TFTP.handler import TftpHandler
+
+ self.session.lm.tftp_handler = TftpHandler(self.session, self.endpoint,


Why create a custom TFTP handler here? This one is created when Tribler starts up.

Because I use custom dispersy instance, and I need to put its endpoint on TFTP instance.

Do you even need TFTP for your experiment? If not, please remove it.

Yes I need that, indirectly. I used RemoteTorrentHandler.download_torrent which called TftpRequester which use TFTP.

devos50 · 2017-01-10T09:26:47Z

experiments/tribler/channel_download.py

+ try:
+ os.makedirs(self.upload_dir_path)
+ except OSError:
+ # race condition of creating shared directory may happen


If you want to prevent this, you should make sure that only one process (client) is creating the directory. Or you can Gumby be responsible for the creation of this directory before you start the experiment.

This condition usually happened in DAS experiment. I couldn't find gumby stuff to handle the directory creation on slave nodes nor I can control which process run in which nodes.

Ah I see the problem. Please document this situation a bit better in the code then please (in the comment) since it might lead to confusion.

Done! If there is no other feedback, I can squash the changes

devos50 · 2017-01-10T09:28:34Z

experiments/tribler/channel_download.py

+ # writing file has not finished yet
+ reactor.callLater(10.0, self.setup_seeder, filename, size)
+ else:
+ # file not found. In DAS case, this is because the file in another node


'because the file is in another node'

devos50

Just saw two more things, sorry for the late review!

devos50 · 2017-01-20T13:38:59Z

gumby/experiments/TriblerDispersyClient.py

+
+ self.session = Session(scfg=self.session_config)
+
+ self.session.start()


This method returns a deferred, please wait for it using either yield or by adding a callback.

I did it a bit differently by now. Check it out

devos50 · 2017-01-20T13:39:05Z

gumby/experiments/TriblerDispersyClient.py

+ self.session.start()
+
+ while not self.session.lm.initComplete:
+ time.sleep(0.5)


Don't use time.sleep.

synctext · 2017-01-25T15:13:01Z

rebase and review ?

Squashed commits : - applying TFTP to download from peers - Merge upload and download ds callback - Make sure dispersy running - Add logger.conf - Force connect peer from Dispersy candidates - Delete data when finish experiment - Enable multi-download peer call - Add availability to logs - Stop stalled download - Reduce download speed - Applying post script and logging - Add name id to distinguish from other similar experiment - Add stop download function - Filter local peers to not included in DownloadState - Fix setup seeder race condition - Add set_speed function - Fix publish swarm if its joined any community already - Set to seed as much as possible in both seeder and leecher

Squashed commits : - Add Multiswarm channel experiment

Squashed commits : - Use Tribler's dispersy only if available - Make endpoint reference - Remove obsolete set_videoplayer

…ents

ardhipoetra · 2017-01-25T17:14:02Z

rebased and ready to review again

ardhipoetra force-pushed the channel_dl branch 22 times, most recently from d79a825 to 31f10e9 Compare November 14, 2016 12:58

ardhipoetra force-pushed the channel_dl branch 3 times, most recently from d0bcc48 to 147962b Compare November 14, 2016 18:58

ardhipoetra force-pushed the channel_dl branch 6 times, most recently from d63afe7 to ebfdf7b Compare November 21, 2016 19:39

ardhipoetra force-pushed the channel_dl branch from ebfdf7b to e2a5065 Compare November 24, 2016 10:36

synctext mentioned this pull request Dec 15, 2016

Operational credit mining branch, Gumby scenario and multichain Tribler/tribler#1842

Closed

ardhipoetra force-pushed the channel_dl branch from 5c62d09 to 014ce2f Compare December 23, 2016 19:41

devos50 requested changes Jan 10, 2017

View reviewed changes

ardhipoetra force-pushed the channel_dl branch 4 times, most recently from 1d64359 to d719e4e Compare January 12, 2017 13:21

devos50 requested changes Jan 20, 2017

View reviewed changes

ardhipoetra force-pushed the channel_dl branch from a673fd0 to a2ad1c1 Compare January 20, 2017 15:35

ardhipoetra added 7 commits January 25, 2017 18:11

Add Channel Download scenarios

ab6e2b0

Squashed commits : - Add Multiswarm channel experiment

Refactor existing experiment to adapt Channel Download experiments

5ada632

Squashed commits : - Use Tribler's dispersy only if available - Make endpoint reference - Remove obsolete set_videoplayer

Add post experiment script

2390791

fixup! Add Channel Download scenarios

e93e10f

fixup! Add Channel Download scenarios

a43e66e

fixup! Refactor existing experiment to adapt Channel Download experim…

0f860dc

…ents

ardhipoetra force-pushed the channel_dl branch from a2ad1c1 to 0f860dc Compare January 25, 2017 17:11

devos50 changed the title ~~WIP : Tribler channel downloading experiment~~ ON HOLD: Tribler channel downloading experiment Feb 2, 2017

devos50 mentioned this pull request Feb 2, 2017

Refactor/rebase the credit mining code when the core functionality is done #296

Closed

qstokkink closed this Sep 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ON HOLD: Tribler channel downloading experiment #282

ON HOLD: Tribler channel downloading experiment #282

ardhipoetra commented Nov 8, 2016 •

edited

tribler-ci commented Nov 8, 2016

synctext commented Nov 8, 2016

synctext commented Nov 8, 2016

synctext commented Nov 8, 2016

ardhipoetra commented Nov 14, 2016

devos50 commented Jan 10, 2017

devos50 left a comment

devos50 Jan 10, 2017

ardhipoetra Jan 10, 2017

devos50 Jan 12, 2017

devos50 Jan 10, 2017

ardhipoetra Jan 10, 2017

devos50 Jan 12, 2017

ardhipoetra Jan 12, 2017 •

edited

devos50 Jan 10, 2017

ardhipoetra Jan 10, 2017

devos50 Jan 12, 2017

ardhipoetra Jan 12, 2017

devos50 Jan 10, 2017

ardhipoetra Jan 10, 2017

devos50 left a comment

devos50 Jan 20, 2017

ardhipoetra Jan 20, 2017

devos50 Jan 20, 2017

ardhipoetra Jan 20, 2017

synctext commented Jan 25, 2017

ardhipoetra commented Jan 25, 2017


		from Tribler.Core.TFTP.handler import TftpHandler

		self.session.lm.tftp_handler = TftpHandler(self.session, self.endpoint,


		self.session = Session(scfg=self.session_config)

		self.session.start()

ON HOLD: Tribler channel downloading experiment #282

ON HOLD: Tribler channel downloading experiment #282

Conversation

ardhipoetra commented Nov 8, 2016 • edited

tribler-ci commented Nov 8, 2016

synctext commented Nov 8, 2016

synctext commented Nov 8, 2016

synctext commented Nov 8, 2016

ardhipoetra commented Nov 14, 2016

devos50 commented Jan 10, 2017

devos50 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ardhipoetra Jan 12, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devos50 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

synctext commented Jan 25, 2017

ardhipoetra commented Jan 25, 2017

ardhipoetra commented Nov 8, 2016 •

edited

ardhipoetra Jan 12, 2017 •

edited