First Credit mining prototype #2064

ardhipoetra · 2016-04-01T18:14:37Z

This PR handles first protoype of auto credit mining in Tribler. Actually, it has been mentioned in #573.

Some GUI components change in home screen. Instead of showing most recent/popular video, list of popular channels is shown. Selecting (or unselect) those checkboxes will affect what channel that we will mine.

A new GUI page called credit mining is there. Basically it didn't change much from #23. I just changed the adding/removing sources part (especially channels) to be done in single click. And now, each source has it's own statistics, and I keep the total statistics in the top part of the screen. Some source info also available in the right part.

More in the core:
When Tribler start the credit mining process, it will (by default) look for swarm that has less seeder ratio compared to all the peers. Tribler will auto-download those swarm, and maximize the ratio credit if possible. We also determined the ratio we want.

Some progress shown in #1842. It includes (also some works that need to be done):

revamped some GUI's (cc80e26)
Fixing known bugs from previous work (415efe5)
Dead tracker reduces performance (blacklisting OR completely integrate with central tracker management)
DHT-only support (not perfect but will do for now) (ec2586b).
Basically, it translates information we can get from peers discovered by DHT to swarm health (seeder/leecher)
Main download (in download screen) aware mechanism (cee0d5d).
We don't want main downloading activity become slower because of mining. Also, continue to download some stuff that already mined.
DHT Performance evaluation -beta (nothing to do with Tribler code)

TODO list on feedbacks :

Core :

called checkpoint twice feedback. Inherited from @whirm branch. I have pointed out (probably) the reason above.
pointless touch channel instance? this. Inherited from @whirm branch.

Test :

Not to boot up complete session in here. I've tried but this is the least I can get it to work.
Use MockObject here. I do have some mock object by my own. It's quite complicated to use lambda so I stick with mine for now.

Other :

twistd plugin on (deleted) boostchannel.py

tribler-ci · 2016-04-01T18:14:39Z

Can one of the admins verify this patch?

devos50 · 2016-04-01T18:16:08Z

add to whitelist

devos50 · 2016-04-01T18:17:06Z

@ardhipoetra nice! You should probably rebase since I see some commits that have been merged into devel already 👍

ardhipoetra · 2016-04-01T19:21:00Z

@devos50 I'm thinking on rebasing after #2038 is merged. I will check something with that part anyway

leahparsuidualc-zz · 2016-04-01T19:51:02Z

Though it isnt the right place to ask, would you point out the most necessary steps to generate this explicit build on ubuntu-based systems? So to say, a generic step-by-step that can be used to test a build like this, for one that has no overview of the given infrastructure but wishes to contribute in terms of testing with usable reports.

I am right now switching forth and back for some days already between Ubuntu Trusty and Xenial in 32- and 64- -bit in different flavors ( Gnome | Mate | Unity | XFCS | LXDE | Cinnamon, mixed in installed variants for the official releases, as also live ones ( w/wo persistence ) and will grow my personal testing grounds to pi- and pi2- based distros in future and add linux mint rosa and latest, as also win10, 8.1 and 8 to the setups; hardware-base will span Core2Duo | Athlon X2 64 | i7 | P4 | pi-arm-v7 and whenever i find the time a galaxy s4-I9505(Krait)|Note3 running diverse CustomROMS (Omni|CM|PAC);

So the detailed question would be:

what is the perfect testing setup, explicitly for Tribler-RELEASE and Tribler-TESTING-Builds (like the one mentionened in here)
how to organize the BUILDS, so (if possible) an automated test-run-batch would deliver results to you developers
how to BUILD and COLLECT RESULTS manually, so you the developers can profit from it?
if possible, can that in anyway combined with the github-account or tribler-name, so it can be found easily afterwards

I hope my request is understandable.

Just to mention it if it helps, space given is 750GB 15k Barracuda SATA|SAS per above listed CPU-Architecture that i am willed and right now spending for these test-|use-cases. (up to 8 Drives are unused/available for Tests)

Please, point me in the right direction or make use of this request to overhaul your knowledge and compress it with precise guides into a as-simple-as-possible-follow-up for the WILLED.

From my side: i'm interested in rolling by your side as long as possible, no time-limit.

Thanks for your interest.

ardhipoetra · 2016-04-05T18:21:43Z

@claudiusraphael, what do you mean by "this explicit build"? are you interested in this particular (credit mining) work? I see your request rather general than specific. I suggest you to create new issue discussing your request

synctext · 2016-04-07T08:35:47Z

@claudiusraphael quite an interesting post! Early in-depth testing of our releases is something we've struggled with for 10 years, really. Ever since the start of Tribler the most difficult task has been reproducing bugs, especially the once that occur only occasionally or in very special cases.

It would be great if you could spend hours trying out all the various features of Tribler manually and on multiple OSes, making you the official Quality Assurance Release Officer :-)

It might take quite some back and forth emailing to reproduce bugs. For example, if you had an outdated version of VLC on your Windows computer, Tribler would give a very peculiar error. With a lowest-level possible 50MByte trace file our chief developer tracked this down to a special place in windows DLL hell. Seems Windows didn't respect DLL loading orders. Tribler could use more testing in general with anti-virus tools and firewall stuff that tries to do smart things and messes with Internet connectivity.

synctext · 2016-04-07T08:37:03Z

@ardhipoetra Impressive work! Solid master thesis material.

whirm · 2016-04-07T17:56:21Z

Though it isnt the right place to ask,

Please, create a new issue and move this discussion there.

I'll try to respond your questions there.

Thanks!

whirm · 2016-04-08T12:02:15Z

@ardhipoetra also, don't waste time on improving the gui as we are ditching it in favor of the new Qt one @devos50 is working on, I would just do the minimal necessary to show your work.

leahparsuidualc-zz · 2016-04-10T20:42:41Z

@ardhipoetra : You asked, whether the guideline for building a specific branch | pull request | Version is aimed at Tribler in general or this specific instance (credit mining);

One as well as the other, should be the correct answer from my side.

As i already tried to describe: i am interested in compiling Tribler builds and (if available) run (hopefully batch ...) scripted Unit-Tests or do manual Testing programmatically, so a result can be gained, that can be of worth to you, the developers, for 'Tribler in General' and for those builds, where i (or you) think they should be tested in much detail as possible, for sure especially those that wake (my) special interest, like the 'credit mining'- -variant.

Why would one do that?

Because i like to learn about the workflow you guys established for tribler and how to automate parts of that to make testing for generic, as also specific builds, easier | possible even for those (that like to help and .. ) that have absolutely no idea what has to be done.
Because i think a very specific build like the credit-mining one given here, should end up finding themselves in general tribler; So it might be helpful to have some testing done, that one (or many) could easily setup and run or manually do step by step, to gather rich results for you, the developers.
Because i wish to see Tribler as the-one-cross-platform-anonymized-torrent-sucker, with tasty features and a heck of possibilities to ruin global players day.

Just as an explanation ...

To answer your question in necessary detail: I'd like to know about the exact differences between the generic tribler and the credit-mining-wip and the instructions that enable one to build | patch | publish results in developer-friendly manner (and learn how to interpret the results in forefront to ease the creation of new issues).

P.s.:
"Btw.: Sorry, that it took some time to get back to this; My youngest (4 month old) daughter is a lil' bit sick right now for nearly 2 weeks already, so it might be that it takes time again to look into github."

@synctext @whirm @ardhipoetra However new Issue opened for the request for testing-environment-setup.

When the mentioned generic build- | patch- | publish- -guidelines will be given, it would be helpful if at this place (because it's the origin of my interest and the original question) some differential and relative steps in comparison to the generic-approach, could be left for future followers and testing-willed, so these can easily find the steps that have to be taken.

ardhipoetra · 2016-04-25T14:36:40Z

@devos50 I just did the rebasing with the current devel branch, can you check it out maybe I have done something silly in my branch?

devos50 · 2016-04-25T17:28:55Z

@ardhipoetra you probably made a mistake during rebase since I see commits made by other team members. Make sure you pick the right commits to replay during the rebase.

ardhipoetra · 2016-04-25T17:58:38Z

@devos50 commits performed by other team members is exist because I continued the work from whirm branch, so there are some commits by him that not merged yet. What do you think about the overall content?

devos50 · 2016-04-27T10:44:09Z

@ardhipoetra ah I see. I will do a code review as soon as possible.

tribler-ci · 2016-05-09T12:26:46Z

Can one of the admins verify this patch?

ardhipoetra · 2016-05-10T20:01:59Z

@devos50, I have fixed some violations in pylint. There's still some, though. Can you take a look at it?

devos50 · 2016-05-10T20:09:13Z

Tribler/Policies/BoostingManager.py

@@ -0,0 +1,1114 @@
+# -*- coding: utf-8 -*-
+# Written by Egbert Bouman, Mihai Capotă, Elric Milon, and Ardhi Putra Pratama H
+# pylint: disable=too-few-public-methods, too-many-instance-attributes


I think these lines should be removed. It's probably not good to specify custom pylint behaviour (we have a nice .pylintrc file for that).

That's what I thought as well. But it's already there before I'm writing the code. But, yeah, will do.

ardhipoetra · 2016-06-27T16:42:01Z

@whirm, The squashing is halfway done. I expect around 10-15 commits at the end. I also separated GUI and Core commit. I will finish this before tomorrow afternoon. Is that okay with you?

synctext · 2016-06-27T18:47:49Z

OK. Tuesday is sort of our last point, as we need to do the experimenral release real soon.

Squashed Commits: - Uniform channel source format - Properly remove credit mining data and sources - Several tweaks in BoostingManager This commit contains: - Add ability to enable/disable source - Read configuration file (boosting.ini) - Store DownloadState data in BoostingManager - Validate source before processing - Enable RSS Source (in CM) with local database This commit also put back checkpoint capabilities in LibTorrentDownloadImpl

Squashed Commits: - Integrate uniform channel source into GUI - Add credit mining manager and list - Integrating gumby with Boostingmanager - Update info to summary panel - Fix bugs in credit mining panel and make prettier popular channel @home - Refreshing-connecting home and Credit Mining Panel - Beautify home channel list -need to expand horizontally- - Change credit mining order in GUI - Remove PostInit in creditmining list - Fix channel source interpretation What this commits do is : - Change BoostingDialog GUI as we don't need channel there - Create CreditMiningPanel as container for credit mining tab - Add Credit Mining option in GUI Utility (lists in tab) - Put CreditMiningPanel in MainFrame - Put popular channels in Home. Directly affected with credit mining - Make current credit mining list to adapt new GUI structure

- Manually fire trackers - Logging peer list - Fix seeding_stats reset bug in credit mining - Add more peers information

- implement "feedback" (random) policy if we can't determine the actual policy - implement get_source_object to get source by flexible input (string/binary) - put DownloadState recent value into sources and update variable related with it.

Also squashed: - Clean and add placeholder in GUI - Refactor code to only consider RSS and directory source to be able to remove

in other words : - Reduce creditmining load at Tribler startup - prioritize other things first - Start credit mining module with a certain condition Squashed commits: - Prevent libtorrent 1.0.9 bug in piece_priority - Add more documentation in BoostingManager - Add logging feature - Enable RSS swarms async download - Update torrent swarm info only when necessary - Use taskmanager whenever possible

Squashed Commits: - Fix resetting statistics data in list - Loading credit mining GUI if necessary - Enabling/Disabling feature credit mining in main - Refactor to move credit mining instance to LaunchManyCore - Remove singleton usage in GUI - Disable credit mining by default on GUI - Refactor GUI code to only provide RAW source

This commit fixes: - some english grammar error in documentation - move etree restriction in torrent_checker - fixes parameter needed in credit mining - Add etree as exception in torrent_checker, add param to force scrape

Squashed commits/fixes : - permit cm pref to be changed in runtime - delete boosting.ini file - Remove unused-obsolete function and fixes function name

Squashed commits : - Separate Credit mining policies in different file - Refactor constant and functions to different file - Removed unused part of the code in utilities - Keep BoostingSource and BoostingManager clean - Extract lambda-variable as function in policies - Refer BoostingManager object in LaunchMany - Separate instantiation and start source Because of this, creating new instance will not make mining automatically start. Calling start function is needed. - Refactor source to have its own TaskManager - Use custom RSS parser instead of libtorrent's

This commits also did some changes in defaults.py: - Specify default configuration - Disable credit mining by default - Remove default channel to boost By this commit, only RSS feed from etree.org left in the default configuration - Remove old write_resume_data function By this commits, use new save_resume_data function - Remove singleton usage in LaunchManyCore

Note for boostchannel.py: We don't have separate GUI and Core yet. This file is standalone program to boost a channel. Will be created later if we have separate GUI and Core.

This remove complex switch policy created in previous commits Squashed commits: - Use translated seed/leech from peers if needed - Add validate source function - Refactor get_source_object to accept single type parameter - Immediately load config in credit mining startup

This is a combination of 12 commits. Squashed commits : - modify testcase to fit new preferences - Remove singleton in test - Change test to adapt TaskManager in source - Add test in dependencies and default load - Move rss xml creation to separate method - Refer channel and torrent creation to RestAPI test - Add more Levenshtein distance test - Add test on custom RSS parser - Refactor test on single type parameter in BoostingManager - Remove threading event usage in test - Reduce timeout in RSS test - Test cleaning pylint

ardhipoetra · 2016-06-28T04:18:06Z

@whirm, @synctext, it's ready now. I trim it up until 16 commits excluding your stuff (8 commits). I rebased it several times, and I think it quite stable.

leahparsuidualc-zz mentioned this pull request Apr 10, 2016

REQUESTS: Description | Example --- Tribler- -testing on various platforms / Guidelines #2078

Closed

whirm added the WIP label Apr 11, 2016

ardhipoetra force-pushed the credit_mining branch from 9949e2c to 35dea9d Compare April 25, 2016 13:57

ardhipoetra force-pushed the credit_mining branch 2 times, most recently from 1d1f3b1 to 972b922 Compare April 25, 2016 17:56

ardhipoetra force-pushed the credit_mining branch 7 times, most recently from 6b096fd to b4fa461 Compare May 10, 2016 19:57

devos50 reviewed May 10, 2016
View reviewed changes

ardhipoetra force-pushed the credit_mining branch from 9bfe48a to d24142d Compare June 27, 2016 15:00

ardhipoetra and others added 5 commits June 28, 2016 00:31

Extract peerlist to its own function

9a87521

- Manually fire trackers - Logging peer list - Fix seeding_stats reset bug in credit mining - Add more peers information

Create function to translate peers into health

6600773

- implement "feedback" (random) policy if we can't determine the actual policy - implement get_source_object to get source by flexible input (string/binary) - put DownloadState recent value into sources and update variable related with it.

Create holder in GUI if credit mining is not ready yet

d7d9ccf

Also squashed: - Clean and add placeholder in GUI - Refactor code to only consider RSS and directory source to be able to remove

ardhipoetra force-pushed the credit_mining branch 3 times, most recently from 7b49767 to f3b870a Compare June 28, 2016 02:24

ardhipoetra force-pushed the credit_mining branch from f3b870a to 6e74ffc Compare June 28, 2016 02:59

Add dirdialog for convenience choosing source

9b9c01a

ardhipoetra force-pushed the credit_mining branch from 6e74ffc to d941d61 Compare June 28, 2016 03:05

ardhipoetra force-pushed the credit_mining branch from 4f3929c to b9d1940 Compare June 28, 2016 03:31

ardhipoetra added 8 commits June 28, 2016 05:31

Clean onexit in BoostingManager

a33b4c7

Add and clean documentation in some functions needed

6cc9d7d

This commit fixes: - some english grammar error in documentation - move etree restriction in torrent_checker - fixes parameter needed in credit mining - Add etree as exception in torrent_checker, add param to force scrape

Add credit mining preferences by SessionConfig

2310579

Squashed commits/fixes : - permit cm pref to be changed in runtime - delete boosting.ini file - Remove unused-obsolete function and fixes function name

Add logger.conf, Delete bootstraptribler.txt and boostchannel.py

57e8636

Note for boostchannel.py: We don't have separate GUI and Core yet. This file is standalone program to boost a channel. Will be created later if we have separate GUI and Core.

ardhipoetra changed the title ~~WIP : First Credit mining prototype~~ READY : First Credit mining prototype Jun 28, 2016

whirm changed the title ~~READY : First Credit mining prototype~~ First Credit mining prototype Jun 28, 2016

whirm merged commit 2f49ace into Tribler:devel Jun 28, 2016

ardhipoetra deleted the credit_mining branch June 29, 2016 06:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First Credit mining prototype #2064

First Credit mining prototype #2064

ardhipoetra commented Apr 1, 2016 •

edited

tribler-ci commented Apr 1, 2016

devos50 commented Apr 1, 2016

devos50 commented Apr 1, 2016

ardhipoetra commented Apr 1, 2016

leahparsuidualc-zz commented Apr 1, 2016

ardhipoetra commented Apr 5, 2016

synctext commented Apr 7, 2016

synctext commented Apr 7, 2016

whirm commented Apr 7, 2016

whirm commented Apr 8, 2016

leahparsuidualc-zz commented Apr 10, 2016

ardhipoetra commented Apr 25, 2016

devos50 commented Apr 25, 2016

ardhipoetra commented Apr 25, 2016

devos50 commented Apr 27, 2016

tribler-ci commented May 9, 2016

ardhipoetra commented May 10, 2016

devos50 May 10, 2016

ardhipoetra May 10, 2016

ardhipoetra commented Jun 27, 2016 •

edited

synctext commented Jun 27, 2016

ardhipoetra commented Jun 28, 2016

First Credit mining prototype #2064

First Credit mining prototype #2064

Conversation

ardhipoetra commented Apr 1, 2016 • edited

TODO list on feedbacks :

tribler-ci commented Apr 1, 2016

devos50 commented Apr 1, 2016

devos50 commented Apr 1, 2016

ardhipoetra commented Apr 1, 2016

leahparsuidualc-zz commented Apr 1, 2016

ardhipoetra commented Apr 5, 2016

synctext commented Apr 7, 2016

synctext commented Apr 7, 2016

whirm commented Apr 7, 2016

whirm commented Apr 8, 2016

leahparsuidualc-zz commented Apr 10, 2016

Why would one do that?

Just as an explanation ...

ardhipoetra commented Apr 25, 2016

devos50 commented Apr 25, 2016

ardhipoetra commented Apr 25, 2016

devos50 commented Apr 27, 2016

tribler-ci commented May 9, 2016

ardhipoetra commented May 10, 2016

devos50 May 10, 2016

Choose a reason for hiding this comment

ardhipoetra May 10, 2016

Choose a reason for hiding this comment

ardhipoetra commented Jun 27, 2016 • edited

synctext commented Jun 27, 2016

ardhipoetra commented Jun 28, 2016

ardhipoetra commented Apr 1, 2016 •

edited

ardhipoetra commented Jun 27, 2016 •

edited