Experiment: one Android phone can reach an audience of millions at no cost

[synctext]

Experiment+storyline for scientific article

Objective: Every Android phone in the world has the potential to become a TV studio, without any infrastructure requirements

Goal: a single hash becomes a real-time updating channel with guaranteed availability
Experiment: A single Android phone is the source of various videos, only identified by a channel hash.
When a new video comes out it is immediately sees and replicated automatically in the network. A new version of Tribler will automatically boost a hardcoded test channel.
Task: boost content available with solid privacy and earn credits

• Android phone can create its own channel and make available in Tribler network (@brussee)
• Android phone scripted in Jenkins and can upload a new video file to channel (@brussee)
• Credit mining operational, hard-coded to boost seeding of Android test phone channel (@ardhipoetra)
• Measure real-time boosting of a new injected video (@ardhipoetra )
• Anonymous tunnels are used in above experiment (@lfdversluis )

[ardhipoetra]

Additional contraint :
Seeding storage should only take 1 GB, at most

[MaChengxin]

Interesting project and I will work on it!

[synctext]

Hacking Lab direction: 20 phone demo. Show how 1 video is recorded and shows up on 20 phones+ able to stream. 6 page IEEE format report with performance analysis. Emulate 100 phones?

ToDo for Friday:

• read stuff
• Understand Minimal viable Android app #2231
• Android app in Android studio with NDK
• test : https://jenkins.tribler.org/job/Build-Tribler_Android-Python/lastStableBuild/

[MaChengxin]

Tips on setting up Tribler Android App development environment on Linux

tested OS: Debian stretch

Step 1: set up dependencies and install p4a

Please read the instructions in the official documentation first, and before setting up please read the following tips as well.
Tip 1: install swig, autoconf, libtool, pkg-config, if they are not already installed (if there are more missing libraries, install them too)
Tip 2: use openjdk-8 instead of openjdk-7
Tip 3: do not install the optional ccache in the first place
Tip 4: do not use pip but rather apt-get to install cython and virtualenv
Tip 5: use pip install --user git+https://github.com/kivy/python-for-android.git (meaning using the latest commit on master from kivy)
Tip 6: In order to prevent Download error: unknown url type: https -- Some packages may not be found! that may occur when building, ssl needs to be enabled for Python. Check this post and this post for instructions. This pull request could also help.

Step 2: install Android Studio with SDK and NDK (from within Android Studio)

From this step on, check the path name before proceeding. chengxin is used as an example here.
Step 2.1: google Android Studio and follow the official instructions
Step 2.2: create symlinks by sudo ln -s /home/chengxin/Android/Sdk/ /opt/android-sdk and sudo ln -s /home/chengxin/Android/Sdk/ndk-bundle /opt/android-ndk

Step 3: set up Git workflow

Step 3.1: fork Paul's repo on GitHub
Step 3.2: clone it recursively (because of the submodules) to the local machine
Step 3.3: set Paul's repo as the upstream
N.B.: when syncing the repositories, note that Paul used force push to GitHub. This post explains how to clean up a fork and restart it from the upstream

Step 4: build the environment (or craft?)

Step 4.1: go to home/chengxin/tribler/android/TriblerService
Step 4.2: run export PATH="~/.local/bin:$PATH" to make sure p4a can be used, run export PATH=/opt/android-sdk/platform-tools:$PATH to add the platform tools, set workspace by export WORKSPACE=/home/chengxin/tribler, otherwise the default value (which is an absolute path) will be used
Step 4.3: run ./build.sh
N.B.: if build fails, before rebuilding it is required to clean the broken builds by p4a clean_builds, and p4a clean_dists

Step 5: open the project in Android Studio

open ~/tribler/android/TriblerApp instead of ~/tribler in Android Studio, and choose Project's SDK when prompted.

[brussee]

N.B.: use pip install --user git+https://github.com/brussee/python-for-android.git

The build now works with the latest commit on master from kivy: kivy/python-for-android@9fa15b7

N.B.: some download links in the default installing process are not working. This might be solved by copying the folder from another PC which has completely downloaded all the stuff, and paste it to ~/.local/share/python-for-andorid/packages.

On Jenkins and my laptop it works with a clean p4a install, see: https://jenkins.tribler.org/job/Build-Tribler_Android-Python/lastStableBuild/

[synctext]

Interesting goal is creating a fully automated measurement and experiment setup.

• automatic uploading of a new .apk
• start .apk on numerous phones, hopefully even 32 phones, probably just simply USB connected
• extract results and parse files
• create results graphs

Extension on that work is doing Android platform emulation, hopefully going beyond 32 phones on our own 200+GByte machines (speed is not a concern). related Android emulations work. Vital to preserve memory

If Android emulation does not provide results within a few days, look at just running Dispersy channel.py.

See https://jenkins.tribler.org/job/Experiment_Demers_devel/557/artifact/output/demers_histogram.png
This shows the exponential spreading of information, should be identical in the smartphone. This experiment has not gotten any attention for over 4 years..

[MaChengxin]

Proposal for the To-Do's of the Hacking Lab Report

• Analyse the architecture of the current project (by reading docs and previous reports, and inspecting the code)
• Conduct some experiments (reproduce and extend)
• Identify the bottlenecks and propose hypothetical solutions
• Reflect on the Software Development Process (such as continuous integration and test automation)

[MaChengxin]

Before running the experiment I read the scripts to understand what is expected to happen. Now I've got some questions for @brussee :
If I understand correctly, in order to reproduce the experiment using, say, 10 real phones, it is required to execute the script for every phone, and then wait for the result. (The script is already copied to the local machine as someScript.sh.)

The questions are:

• Is my understanding correct? If yes I will take a closer look at the experiment.sh file to understand what it does, especially the command after echo Start experiment;
• Is the figure in your thesis report automatically generated after running the experiment? It looks like that adb_pull.py will take care of getting the experiment result, but it is not responsible for plotting. So what did you use to plot the figure for your report?

[MaChengxin]

A question for @devos50
After I pushed a commit to GitHub, Jenkins does not start building as expected. "Build when a change is pushed to GitHub" is ticked in Build Triggers, but there is no new build. Is that because the server is turned off?

[devos50]

@MaChengxin you need to configure a web hook in your Github repo for that function to work (see https://help.github.com/articles/about-webhooks/). Let me know when you need any help to set that up!

[MaChengxin]

@devos50 Thanks!
The current status is that Jenkins (GitHub plugin) is set up but the hook has never been triggered, as indicated by the gray dot:

However, I can see that the GitHub Hook Log is changed after I click the "Test Service" button on GitHub.

Another noticeable thing is that a build (#2, manually terminated) was once triggered when I was setting up the hook, but I am not sure if that is a coincidence or I set it up correctly. The Jenkins hook URL I am using is https://jenkins.tribler.org/github-webhook/. Is it right?

Full setting:

[synctext]

is this perhaps an authorization problem?

[devos50]

@MaChengxin it seems that the build has been triggered with a GitHub push? https://jenkins.tribler.org/job/pers/job/Build-Tribler_Android-Python_Chengxin/2/pollingLog/

[MaChengxin]

@devos50 Yes, that's true, I checked the commit id in the log and it is the one I made. But I am just curious about that:

• why on GitHub it still says "This hook has never been triggered"?
• why no result was shown after I click the "Test Service" button? I was expecting something like "Test Passed/Failed", but I only saw "Okay, the test payload is on its way."

UPDATE (Dec 5, 15:41): Jenkins is now able to track the changes on GitHub, after setting the Branch Specifier to be refs/heads/android-app as suggested by Jenkins Hint.
The weird thing is that on GitHub it still says "This hook has never been triggered", but I will leave it like this.
I created an empty file test-jenkins.md and pushed to origin, and then Build 8 was triggered;
After that, I revoked the commit on GitHub by git push -f origin HEAD^:android-app, and this led to Build 9.

[brussee]

Is my understanding correct? If yes I will take a closer look at the experiment.sh file to understand what it does, especially the command after echo Start experiment;

That is the Multichain scalability experiment. This is what you need: https://github.com/brussee/tribler/tree/android-app-demo-2/android/TriblerApp/scripts It is called demo_time.sh

Is the figure in your thesis report automatically generated after running the experiment? It looks like that adb_pull.py will take care of getting the experiment result, but it is not responsible for plotting. So what did you use to plot the figure for your report?

The results are processed with Matlab: https://github.com/brussee/thesis/tree/master/phone-to-phone/run3

[MaChengxin]

@brussee
I tried to reproduce the experiment using 3 phones but failed.
First of all, I looked at the script demo_time.sh and executed the commands in it for Phone 6, and got an empty file phone6.dat. This is not surprising as the log could be removed after the last experiment.
Then I recorded a video, created a channel, and published it (by clicking yes when prompted). However, I did not see anything on the other two phones. Are there any steps missing here?
Then I tried to run prep_demo.sh but stuck at the final step, because I do not have Tribler_appstate/no_favs_no_id/.Tribler on my machine. How is this file/folder generated?

[brussee]

Make sure the phones are subscribed to your new channel and have it open on screen.
You can use any app state you want, but remember if you use a clean slate it may influence your results because a lot of syncing will happen for AllChannel.
I used a common appstate from running Tribler for a few days and removed the private keys, so Tribler creates a unique ID for each phone.

[MaChengxin]

Make sure the phones are subscribed to your new channel and have it open on screen.

For this, I did the simplest test with two phones, one being the source (Phone A, Galaxy S6) and the other one being the subscriber (Phone B, Nexus 6). Then I observed:

• A channel created on Phone A will not show up immediately on Phone B after Phone B subscribes it; the channel will show up when a video is sent via this channel;
• If Phone B has already received a channel from Phone A, renaming the channel on Phone A will not take effect on Phone B, even after removing the channel and receiving again;
• It will take a long time before a received video can be played, even if it is quite small (1.2 MB);
• After removing a received channel on Phone B, the video list on Phone A will disappear from the channel;
• After reinstalling the app, the list of favorite channels is not removed (I still see "S3" on Galaxy S6);
• Sometimes favorite channels cannot be removed (it will show up again);

The following are my questions: @brussee

1. Is my observation caused by that I did not operate properly, or simply the bugs of the app?
2. How many channels can be created on one phone? (Looks like only one.)
3. What does "Send to nearby phone" send? Is it only the info of a channel or videos in that channel?

[brussee]

1. It takes a while to sync channel information (creating and editing, also manual gui refresh required).
   Adding (auto refresh) and removing videos (manual refresh) is near instant if subscribed to that channel.
   App state is stored separately on the (emulated) sdcard and is not removed if the app is uninstalled for easy debugging.
2. Only one channel can be your own channel at a time.
3. The dispersy channel id is send in plain text.

[MaChengxin]

@brussee :

1. My understanding of an "app state" is that it records the activity of the app, and some data (such as the synced channels) as well. By pushing to other phones, these phones could be deceived that they have also done those activities even if Tribler is just clean installed on them. Is that right?
2. Does a private key in an app state record the info of the current phone? How to remove it? (I checked the .Tribler folder but did not find anything similar to it.)
3. Where does the file holding channel information reside? I wanna take a look at it to compare a channel originally created and one received from other phones.

[brussee]

1. Yes. The app state (.Tribler) is only related to the back-end service (Tribler core) and includes databases etc.
2. The private key is used to sign messages exchanged in communities. Remove (backup first) the four *.pem files in .Tribler.
3. .Triber/sqlite/tribler.sdb in table "_Channels"

[MaChengxin]

Thanks! @brussee
The (manual) experiment with two phones went well and the weird bug (Item 4 in my comment of yesterday) was gone.
Now I've got the following experiment data from the subscriber phone.

12-08 15:21:44.974 V/AddedToChannel( 8021): VID_2016-12-08_15_21_27.mp4 dispersy_cid: 89dbd4cc8ed434f41a5b8b8901eca2d6e8bc7d38 (a3c82f695a78dfa99e64103fee9b31a93b3b699e)
12-08 15:21:44.975 V/AddedToChannel( 8021): VID_2016-12-08_15_21_27.mp4 dispersy_cid: 89dbd4cc8ed434f41a5b8b8901eca2d6e8bc7d38 (a3c82f695a78dfa99e64103fee9b31a93b3b699e)
12-08 15:23:20.508 V/AddedToChannel( 8021): VID_2016-12-08_15_23_03.mp4 dispersy_cid: 89dbd4cc8ed434f41a5b8b8901eca2d6e8bc7d38 (3de9551c5d6c3655e16e88f83502326df5bd26b2)
12-08 15:23:20.510 V/AddedToChannel( 8021): VID_2016-12-08_15_23_03.mp4 dispersy_cid: 89dbd4cc8ed434f41a5b8b8901eca2d6e8bc7d38 (3de9551c5d6c3655e16e88f83502326df5bd26b2)
12-08 15:23:20.511 V/AddedToChannel( 8021): VID_2016-12-08_15_23_03.mp4 dispersy_cid: 89dbd4cc8ed434f41a5b8b8901eca2d6e8bc7d38 (3de9551c5d6c3655e16e88f83502326df5bd26b2)
12-08 15:25:55.289 V/AddedToChannel( 8021): VID_2016-12-08_15_25_36.mp4 dispersy_cid: 89dbd4cc8ed434f41a5b8b8901eca2d6e8bc7d38 (2c6f74fd16121630667428092c9cd5d84397ea83)
12-08 15:25:55.296 V/AddedToChannel( 8021): VID_2016-12-08_15_25_36.mp4 dispersy_cid: 89dbd4cc8ed434f41a5b8b8901eca2d6e8bc7d38 (2c6f74fd16121630667428092c9cd5d84397ea83)
12-08 15:25:55.298 V/AddedToChannel( 8021): VID_2016-12-08_15_25_36.mp4 dispersy_cid: 89dbd4cc8ed434f41a5b8b8901eca2d6e8bc7d38 (2c6f74fd16121630667428092c9cd5d84397ea83)

1. Why is video VID_2016-12-08_15_21_27.mp4 added to the channel twice? (Other videos even three times)
2. In the first line, is "12-08 15:21:44.974" the time a video be added to the source phone's channel, or it is the time when discovered by the subscriber?
3. What does the strings in the parentheses after the string of dispersy_cid stand for?

[brussee]

1. That should not happen.
2. When the subscriber phone shows it in the GUI, right after discovering the content.
3. Infohash of the torrent.

Also, take a look at the source code of the app. You can find the answers to the last two questions there.

[MaChengxin]

@brussee

1. I adapted demo_time.sh a little bit to get the log from the source phone. What I did is changinggrep "V/AddedToChannel" to grep "V/addTorrent" to get the time when a video's torrent is added, which is used in parse_data.m. I guess you used an alternative way to redirect the log into the file, since it has more information. Did you also document the script?

2. After adding 3 videos into the channel, the log of phone 1 through 4 (the subscribers) is something like:

12-15 16:08:03.130 V/AddedToChannel( 8661): VID_2016-12-15_16_07_39.mp4 dispersy_cid: e345f5be2ec5f8af01a1c374ead2d665552123ba (2001c1c974dfceb0df2a09da08df0ba01a091e74)
12-15 16:08:43.122 V/AddedToChannel( 8661): VID_2016-12-15_16_08_29.mp4 dispersy_cid: e345f5be2ec5f8af01a1c374ead2d665552123ba (ccf6405d0b9244c31dc306b5049a5807703a216d)
12-15 16:09:08.252 V/AddedToChannel( 8661): VID_2016-12-15_16_08_55.mp4 dispersy_cid: e345f5be2ec5f8af01a1c374ead2d665552123ba (9dfb029fe148696fb4c9fbe8da650efc01cbaea9)

Yet on the source phone only one video is logged:

12-15 16:09:06.631 V/addTorrent( 3770): Torrent is now added!

Did you also meet this problem? If yes, what did you do to circumvent the mismatch of numbers of videos?

[brussee]

1. I did that by hand.
2. No, see 1.

[MaChengxin]

@brussee
I am trying to reproduce the experiment of Multichain performance but unfortunately it did not work.

On the one hand, I have some questions about this experiment in your thesis (Section 6.2).

1. It says that Multichain signs a block every 10 minutes, but in Figure 6.4 the blocks are created in a much higher speed. For instance, on the laptop 25000 blocks are generated in less than 2000 seconds. How can it be so fast?
2. The caption of the figure says "creating and signing of 25000 blocks between two peers". Can you elaborate on "between two peers"? Isn't the node existing in the entire network?

On the other hand, I am not sure if I have set up the experiment correctly.
Is it as simple as executing the experiment.sh after turning on Tribler on an Android device which is connected to the laptop? I tried this script but it won't stop. Are there any missing steps?

PS: Have you meet a situation that the Tribler app would behave abnormally after rebuilding it after a Debian update? In my case Debian is upgraded to 9.0, building Tribler was OK but the app would be stuck at starting service.

[brussee]

1. Normal operation is every 10 minutes. This experiment is a performance experiment, so it is going as fast as possible by simulating traffic.
2. See 1. it is simulating traffic. Take a look at the source code: https://github.com/brussee/tribler/blob/d1a1d0ba08ba4a3fca9ef99aad3e74859a7454e3/android/TriblerService/service/ExperimentMultiChainScale.py#L27

The experiment would always work for me, but the automated result fetching sometimes does not work. Use adb pull to get "ExperimentMultiChainScale.dat" manually from the device or run the last few lines of the script by hand.

If you do a distribution upgrade of the build system all bets are off. I suggest only doing that if you have a good reason to do so.

[MaChengxin]

@brussee
Thanks for the quick reply. I will look into the code next week.
The OS version was accidently upgraded when I did a sudo apt-get update/upgrade and it caused so much trouble. I am considering rolling back from Debian testing to stable.

[synctext]

Brainstorm alternative thesis direction (backup plan): validated identities using phone numbers and SMS phone number validation or optionally validation using government IDs. use hashing. web-of-trust and sybil-proofness.

[MaChengxin]

@synctext
Can you please elaborate on it? Or we discuss it during our next meeting.

[MaChengxin]

@brussee
The Multichain experiment still did not go as expected so I've decided to skip it for a which and proceed with the following experiments. 6.3 Startup time and 6.4 Content creation were pretty simple and I am now doing 6.5 API responsiveness.
I understand that the API responsiveness is tested by accessing 127.0.0.1/channels/discovered 1000 times through port 8085 on the device. As JMeter runs on the development machine we need to forward the command to the device.
I found the usage of adb forward here, but failed to do adb forward tcp:5037 tcp:8085, which gives an internal error. (I chose 5037 as "all adb clients use port 5037 to communicate with the adb server" according to the doc.)
So how did you forward the API port to make the experiment run correctly?

[brussee]

I found the usage of adb forward here, but failed to do adb forward tcp:5037 tcp:8085, which gives an internal error. (I chose 5037 as "all adb clients use port 5037 to communicate with the adb server" according to the doc.)

So exactly that port is already in use (by adb itself). Simply use adb forward tcp:8085 tcp:8085 and enter 127.0.0.1:8085 into JMeter.

[synctext]

swarm size is still most interesting thesis direction.

[MaChengxin]

@brussee
OK that's my bad. I misunderstood the usages of the ports.
After the modification it works now: https://github.com/MaChengxin/thesis/commit/c0038e48da9d6cfbdb0f39832c226cd339a0272f

[MaChengxin]

@brussee
Now I've moved to 6.6 Profiling.
What is the entry point of starting the experiment? I did some search online only to find examples of profiling single Python scripts instead of a complex Python project, especially when it is running on Android.

[brussee]

After the app is started normally. And I think I used the standard app state that had been idling for a couple of days. Otherwise right after first starting the app with a fresh installation (empty db).

[MaChengxin]

@brussee
Well, what I meant was that cProfile is usually used to profile a Python script running on the same machine, then how do we do profiling on another device (the Android phone)?
It looks that the cprofiler.sh only starts the Twisted daemon, which seems having nothing to do with cProfile. Besides this file, I did not find any other script about profiling.
Which script/command am I supposed to execute in order to do profiling?

[vandenheuvel]

Have you noticed you can profile (at least on linux) with twistd --profile=output.dump -n tribler?

[MaChengxin]

@vandenheuvel
Thanks for pointing out. I didn't know that until today. I also tried this command and it worked.
However, when I was trying to do that for Android, the twistd Activity is nowhere to be found:

chengxin@HP:~/tribler/android/TriblerApp/scripts$ ./cprofiler.sh
Clean logcat
Start twistd
Starting: Intent { cmp=org.tribler.android/.TwistdActivity }
Error type 3
Error: Activity class {org.tribler.android/org.tribler.android.TwistdActivity} does not exist.

What did I do wrong this time? @brussee

[brussee]

TwistdActivity has been renamed to ProfilerActivity

[synctext]

Possible storyline for HackingLab final report. IEEE 2-column format. Focus on 1 experiment:

• We created a fully automated measurement and experiment setup
• end-to-end integration testing
• It can handle numerous Android phones
• The whole experimental chain is fully automated
  • automatic uploading of a new .apk
  • start .apk on numerous phones, hopefully even 20-32 phones
  • extract results and parse files
  • create results graphs
• Android experiment tests:
  • channel subscriptions
  • peer discovery (others within channel, using Dispersy)
  • content creation on Android (1 channel/source phone, create torrent+add to channel)
  • content discovery (new channel content discovery time)
  • Android-to-Android video downloading (auto-download all new content in a channel)

Graphs:

• Channel discovery time (how long does it take to find others in a channel)
• Content discovery time (How long does it take after content is put on the channel, before it reaches various phones)
• Download time (start download till completely downloaded; upload phone bottleneck?)

Possible sections (citation to various papers, including Martijn work+Brussee):

• introduction
• problem description (phone-to-phone networking, video dissemination without servers)
• architecture (software pictures, P4A, tribler,.. {copy pictures+cite others})
• experimental setup (lot of phone)
• Video Dissemination Experiment (3 magic graphs)
• Conclusion (it works and can detect bugs)

[MaChengxin]

@synctext
I found that automating the entire experimental chain might not be of top priority at the moment because I've experienced so many crashes of the app. As long as the app crashes, the automation chain will not make sense because I need to manually restart the experiment at the right point.

[MaChengxin]

@synctext
I made a draft of the report: https://www.overleaf.com/read/gsvbmqqzqdfd
The phone-to-phone downloading functionality isn't working on Android so the experiment can not proceed. I've thought about fixing it but I am not sure how long it would take.

[synctext]

Next possible step: "attack-resilient content popularity estimation for distributed multimedia search"

[brussee]

On section C: My findings indicated that discovery time is heavily dependant on the amount of phones used in the experiment (>10), due to the limit of 10 connected peers at the same time. This may very well be the reason why your measurements, from the 3rd video onwards, diverge less.
On section D: Only video-downloads with a, preferably hardware, supported codec are supported by the app.
On the bugs: Try the lastest stable build of the app on Jenkins. I do not experience the bugs you mention.

[synctext]

Conclusion: app is not stable
This is required to be fixed before a release, only stable on Android phone of Paul it seems.

@brussee if you have time... we have several new Google Pixel to play with...

Final report for Hacking Lab by Chengxin Ma: performance-analysis-tribler.pdf

[synctext]

update:
In 2018 we started work on "Giga channels", channels of magnet links downloaded using Libtorrent. This means we can use congestion control of UDP for fast download of BEP9 magnet links in a special formatted text file. Channel updates are appended.

[synctext]

Android development is painfully slow. We first need to hit the epic milestone of a million users.