Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Universal communication using imperfect hardware #4827

Closed
synctext opened this issue Sep 9, 2019 · 49 comments
Closed

Universal communication using imperfect hardware #4827

synctext opened this issue Sep 9, 2019 · 49 comments
Assignees
Milestone

Comments

@synctext
Copy link
Member

synctext commented Sep 9, 2019

placeholder issue for thesis/survey

Skeleton GUI + 30-ish citation paper.
Example from few years ago: "Survey of robust and resilient social media tools on Android"
https://www.google.nl/search?q=friend+to+friend+torrent
Switch between Libtorrent-broadcast and direct QUIC based transport to your friends
Record protests, broadcast to the world. Please investigate: Hong Kong protesters using Bluetooth Bridgefy app. Plus Life in an Internet Shutdown: Crossing Borders for Email and Contraband SIM Cards. Our own proposal from pre-Snowden days, 7 years ago: https://tools.ietf.org/html/draft-pouwelse-censorfree-scenarios-02
Explicit offline/bluetooth,private and open Internet status:

@synctext
Copy link
Member Author

synctext commented Sep 10, 2019

Open questions:

  • focus on scientific challenge of anonymous spreading or any media usage
  • focus on engineering and friction-less usage on even the lowest-end devices (e.g. Venezuela income levels)
  • social focus: build a decentral social graph. See Scalability & Paranoia in a Decentralized Social Network and our own running code
  • evaluate first prototype with real vulnerable protesters and improve with their requirements (e.g. user-driven methodology)

Possible thesis scope:

  • low-end (< $100) smartphone (e.g. no Python inefficiencies, recommend: Java or Rust language)
  • record protest video
  • disseminate protest videos
    • offline, share with friends
    • share magnet link by email
    • seed in Bittorrent
    • share to CloudoMate or Sporestack
    • anonymous Tribler seeding protocol (optional)
  • discover protest videos
    • offline, receive from friends
    • get a blob of magnet links
    • use undocumented Gigachannel protocol
  • view any format protest video without additional installs (e.g. VLC bundeling)

@synctext
Copy link
Member Author

@synctext
Copy link
Member Author

proof-of-principle inside the survey with 30+ citations and screenshots (IEEE 2-column). Screenshots of prototype V0.013?
science Can we build a store-and-forward based mesh network within the Android security framework? This would service as the principle building block for decentral friend-to-friend network. (Rust?, GPL-openness Bridgefy
SDK?)

@MattSkala
Copy link
Contributor

Current survey topic: Smartphone ad hoc networks (SPANs)

Up-to-date reading list, notes + latex draft: https://github.com/MattSkala/meshnet-survey
Running code: https://github.com/MattSkala/meshnet-android

Current sprint goal:

  • implement device-to-device messaging using Bluetooth, Bluetooth LE, WiFi Direct, WiFi Aware, Nearby Connections API
  • a single interface, multiple transport layer implementations
  • to get familiar with the wireless communication technologies and limitations of Android

Related papers:

@synctext
Copy link
Member Author

Great progress! Screenshot?

@MattSkala
Copy link
Contributor

UI not that interesting, but Nearby Connections and Bluetooth transports are working.

@synctext
Copy link
Member Author

synctext commented Oct 2, 2019

@MattSkala
Copy link
Contributor

Last sprint:

  • WiFi Direct and WiFi Aware exploration/implementation

Currently implemented communication protocols:

(Have not been able to test BLE L2CAP and WiFi Aware yet due to the lack of supporting devices)

Next sprint:

  • focus more on actual writing
  • test with more than 2 devices

trustchain-android could be useful after some refactoring (the core logic should be extracted into a reusable Java library)

@synctext
Copy link
Member Author

synctext commented Oct 15, 2019

Emphasis on science: any-time any-place communication, No-Internet-Needed communication paradigm. Talked about for 40-years, no significant deployment yet. Finally the end-user has software access to the advanced mesh-networking capable devices on affordable devices.
research question.
For appealing writing: No-Internet-Needed science, state-of-the-art (screenshots of barely useful apps), underlying wireless technologies, Operating System Support, Our "MultiConnect" proof-of-principle, etc.
Thesis, live mesh, like we did previously?
img_20180123_114422

@MattSkala
Copy link
Contributor

Testing setup. Each device connected to two other devices, all of them able to communicate with each other using a simple flooding-based mesh network.

IMG_20000101_150509

@synctext
Copy link
Member Author

synctext commented Oct 31, 2019

Impressive prototype, operational ad-hoc networking and broadcasting. No crypto yet in this early work. messaging works! Ad-hoc secure connection fabric

Discussed thesis direction potential: universal any-medium connector and bandwidth accounting (carrier-grade puncturing,freeriding,Trustchain recording; 10k open sockets to puncture symmetric NATs). Any phone as a Internet-relay. "collaborative wireless Internet". 'We are the Internetz"
stripped IPv8 @ Kotlin? and/or Binary blob transfer on top of UDP in-scope?

@devos50
Copy link
Contributor

devos50 commented Nov 2, 2019

Quite related excerpt from the Redecentralized digest:

Meanwhile, in Catalonia, pro-independence group Tsunami Democràtic created an Android app based on Retroshare, the peer-to-peer communication and file sharing software that feels nearly old enough to deserve its name. It’s used in protest coordination, and is currently not even available for iOS nor published on Google Play: people have to download the app as an APK file from its website.

Ironically, GitHub then took the app down for visitors from Spain, after an order from Spanish law enforcement — which is novelty by itself, Spain now being the third country on GitHub’s takedown log, besides Russia and China. But at least, unlike the very centralised landscape of app stores, alternative web hosting options are plenty; and files can be passed around in many ways.

Retroshare seems to align with the goal of making a peer-to-peer social network overlay. It support chatting, mail, file sharing, forums, link sharing and even channel functionality. There even seems to be an experimental VOIP plugin for video and audio chatting.

@synctext
Copy link
Member Author

@MattSkala
Copy link
Contributor

The Internet has been shut down for 120 days in Kashmir, WhatsApp deleting inactive accounts: https://www.buzzfeednews.com/article/pranavdixit/hundreds-of-kashmiris-are-disappearing-from-their-whatsapp

@synctext
Copy link
Member Author

synctext commented Dec 6, 2019

How Facebook Avoids Ad Blockers
The next technique, this is the point where it gets particularly interesting, Facebook actually hides 'dummy' DOM nodes between the 'Sponsored' text. These values are entirely random characters, with a random number of DOM nodes between them. Invisible characters.
image

@MattSkala
Copy link
Contributor

Twitter is funding a small team of researchers to build an “open and decentralized standard for social media,” with the goal of making Twitter a client for that standard.

https://www.theverge.com/2019/12/11/21010856/twitter-jack-dorsey-bluesky-decentralized-social-network-research-moderation

@MattSkala
Copy link
Contributor

Current thesis proposal:

Multi-transport Peer-to-peer Communication Protocol for Mobile Devices

  • Task: Develop a P2P messaging app that is able to communicate
    • with remote peers over the Internet without using any central server,
    • with nearby devices over Bluetooth LE
  • Fundamental problems:
    • Not all peers are connectable (e.g. 2 peers both behind symmetric NATs)
    • Peers connected over Bluetooth should be able to communicate with peers connected over the Internet
    • => There should be a way to route messages in the network. Some peers may act as relays and route others' traffic
    • Optional: What if a target peer is not reachable at the moment? Delay-tolerant network routing could be employed. Messages can be stored by common online friends or random peers with proper incentivization.
    • Optional: To prevent free-riding, the network could take advantage of MB bandwidth tokens built on top of TrustChain. The sender establishes the route and sends MB to all peers along the route. The MB balance can be negative, but peers prefer routing messages of peers with higher MB balance.
  • Peer discovery:
    • Peers are addressed by their public keys
    • Peers are discovered using a bootstrap server, LAN discovery, or Bluetooth LE scan
    • To establish a friendship, both peers have to exchange public keys (e.g. scan each other’s QR code or share an invitation link) to protect against impersonation and MITM attack
    • DHT for resolving public keys to physical IP address (IPv8 style)
  • Routing:
    • First, try to establish a direct connection using DHT and NAT puncturing
    • If a direct connection fails, try to find a path using a routing algorithm
  • App functionality:
    • Send text messages
    • All communication should be E2E encrypted
    • Optional: Send images, videos? (Problem: Binary transfer over UDP – TFTP/QUIC)
    • Optional: Create a chat group involving multiple peers (Problem: How to encrypt group conversations?)
    • Optional: Unencrypted broadcast to all nearby devices with flooding (Bridgefy style)
  • Outcomes:
    • IPv8-compatible client in Java/Kotlin (without dependency on Android SDK)
    • Reusable library for P2P communication on Android with integrated NAT puncturing, Bluetooth mesh networking, message routing based on public key identifiers (built on top of IPv8)
    • Censorship-resilient P2P messaging app for Android (demonstrating the usage of the library)
  • Implementation plan:
    • Take ideas from trustchain-android, but develop in an independent, more maintainable and modular codebase
    • Java vs Kotlin? Java: low barrier of entry for students, Kotlin: future-proof, coroutines (easier multithreading), stronger type system (null safety)

@synctext
Copy link
Member Author

synctext commented Jan 2, 2020

Turkey: arrest of 75,000 suspects, primarily for downloading ByLock app.
India Internet shutdowns: https://internetshutdowns.in/ + HN discussion

@synctext
Copy link
Member Author

synctext commented Jan 28, 2020

Always On Overlay: Universal Connectivity using imperfect hardware

  • FOCUS: Will find a connection between peers. Any wireless hardware or even a proxy;
  • seamless file transfer (expands IPv8!)
  • e2e testing with pre-paid SIMS+phones?
  • offline support, nearby devices
  • bandwidth accounting and cooperation
    • sustainable commons
    • zero-maintenance
    • zero-cost
  • NAT puncturing DHT service?
  • support popular services as simple demo:
    • smartphone-only DirectConnect
      image
    • magnet link download
    • social network explore
    • share binary blob
  • enable disruption of Facebook:
    • needs to be free
    • self-organising
    • The storage&data problem
    • storing the data, getting it from Facebook (e.g. GDPR)

@synctext
Copy link
Member Author

synctext commented Feb 6, 2020

Related work for decentral social network. You need to obtain the public key of your friends within a web-of-trust.

@synctext
Copy link
Member Author

Remarks:

  • private keys should be in cold storage, but public key is now our DHT-routed identity; (fixing this is future work)
    • no cold storage
    • no key revocation
    • no theft locking
  • Friendship certificates and MITM-proof key exchange: good stuff; please limit to 1 week.
  • Current sprint
    • submit PR to main IPv8 documentation, prevent forking
    • trustchain; almost fully compatible
    • bootstrap server issues
    • re-using XML GUI of prior app
  • Next sprints
    • DHT
    • Bluetooth
  • advise on methodology: first implement a ugly-partially-works version and integrate that

@MattSkala
Copy link
Contributor

MattSkala commented Mar 4, 2020

Progress:

@synctext
Copy link
Member Author

synctext commented Mar 4, 2020

Sprint feedback:

  • nice going !!!
    • Multihoming: 4G + bluetooth is operational
    • simply broadcast on available networks for now
    • fix unicode encoding issue. 🦄 works, 😱 does not :-)
  • Denial-of-Service attack on self with 100k UDP packets (3 servers have over 415k blocks)
    • [1 .. -1] formatting of crawl request
    • killing the app does not work, alter your IPv4 to stop it
  • remember to discuss security in your thesis
  • possible directions; next sprintS
    • Thesis
      • "Always On Overlay: Universal Connectivity using imperfect hardware"
      • TOC: introduction, problem description, design, implementation, experiment. Re-usage
      • start writing 3 pages around your screen-shots and bluetooth? or intro?
    • buy 3 pre-paid SIM cards for now (<100Euro please). Next steps is 12+ EU-wide/international ones with solid GByte limits and extended validity.
    • carrier-grade NAT analysis and puncturing
    • UDP volunteer proxy relaying for universal connectivity (e.g. measure of last resort; bandwidth accounting)
    • connect to a known peer: DHT lookup with NAT puncturing (or global database of IPs that magically scales:-)
    • trivial binary transfer or elegant UDP socket re-usage with Libtorrent/LibUTP (take&send picture demo)
    • Friendship certificates and MITM-proof key exchange: good stuff; please limit to 1 week.
    • bluetooth mesh network (cool, but not for this thesis please)
  • https://github.com/Tribler/trustchain-superapp

@MattSkala
Copy link
Contributor

MattSkala commented Mar 16, 2020

Update:

  • project split into kotlin-ipv8 (protocol library) and trustchain-superapp (app)
  • fixed UTF-8 encoding issue (string length was encoded incorrectly for multi-byte characters)
  • TrustChain Super App APK
  • purchased pre-paid SIMs from 3 major operators
    • KPN and T-Mobile both have port-restricted cone NAT, usable for P2P after hole punching
    • Vodafone changes port mapping for new connections every 60 s
      • ports are assigned at random from the range of 45k-55k, port prediction not possible
      • multi hole punching not easy to implement on Android as Java does not allow to change packet TTL
      • possible workaround: connect to as many peers as possible within the first minute, but after that we would need to connect to a new public peer (or a peer behind full/address-restricted cone NAT) to learn the new port mapping
    • for guaranteed connectivity, we need relays with bandwidth accounting
  • created a repo for the report, preliminary TOC, WIP: https://github.com/MattSkala/msc-thesis/blob/master/report/report.pdf

@ProphetZarquon
Copy link

Fascinating research here!
I am convinced that censorship resistant internet-independent decentralized communication apps will never reach a geographic density of peers sufficient to ensure connectivity throughout urban (much less rural or wilderness) areas until cellular or other 1mi+ bands can be leveraged by said app. Local mesh connectivity is all well & good for research & in limited use cases where protests are localized or longer range centralized subscription service is available... but unfortunately the need for unlicensed 1mi+ ranges seems obvious, if free communication is to be assured.

I don't know what to suggest, as even LTE-Direct & LTE-V protocols do not permit connections without permission from a carrier tower, & cellular modems in commonly available end-user devices would require homebrew firmware to bypass such restrictions (or to work as SDR in xmit mode); I'm quite certain longer ranges are needed to make decentralized communication functional enough to achieve general viability though.

Please keep working on these projects (I hope my interjection is not unwelcome!) & know that I applaud your efforts... I am just tired of seeing one project after another languish in isolated deployments & low functional adoption. I believe that (whether legally or not) cellular or other bands with 1mi+ range must be leveraged in order to escape communications captivity.

Without specifically advocating such a course of action, I'd like to hear informed thoughts on this subject?

@synctext
Copy link
Member Author

synctext commented Apr 1, 2020

ports are assigned at random from the range of 45k-55k, port prediction not possible

  • DDoS protection?
  • Really impossible? 1000 open UDP listen ports, 1 MByte per second with 100 Byte packets = 10k packets per second. Will it luckily connect within 1 hour? (brute force/Casino method)
  • "crazy NAT puncturing" focus; later do relaying as last resort (then focus afterwards on bandwidth tokens)
  • SIM infrastructure: € 20 Euro in total
  • DAO use-case: a company in a smartphone
    • show how to apply Universal Connectivity
    • work by students and SuperApp application
    • offer universal code execution + generic economic activity on top of Universal Connectivity

@synctext synctext changed the title Survey on distributed social media (posts,streaming,messages) Universal connectivity using imperfect hardware Apr 1, 2020
@devos50
Copy link
Contributor

devos50 commented Apr 1, 2020

This hackaton project uses an Android app and Bluetooth features to check for people with COVID-19 in your vicinity. From the website:

WeTrace uses Bluetooth LE to keep track of people you were close to. It automatically collects the ids of other users while it shares its own ID to them. When a user is tested positive, he or she can report the status change. A silent notification is broadcasted to all the users. That then can locally compare the newly reported case with their database of collected user IDs. If the reported ID is found in the database, the user that he or she might be at risk - the app will inform the user.

@synctext
Copy link
Member Author

synctext commented Apr 3, 2020

@ProphetZarquon Yes, you indeed need long-hail links if you want to build a useful network. It seems already 8 years ago I did an IETF Internet Standard draft writeup on kill-switches from governments and what to do against them. Moving information around on smartphones, store-and-forward, seems like the only viable method. Directional antenna will only get you so far. Maybe in another 10 years we will have useful soft beamforming MIMO in our software radios in each phone...

@ProphetZarquon
Copy link

ProphetZarquon commented Apr 6, 2020

Without robust decentralized long-range communication options, I think ten years from now anyone knowledgeable enough to establish such solutions will be too scared to even discuss it. Already half almost all the projects I've seen showing promise have gone dark, & while I'd like to believe that's because the people involved went underground, instead I suspect they either took suppression-industry jobs or were put in the ground permanently.

@MattSkala
Copy link
Contributor

  • NAT puncturing of Vodafone NAT
    • Sent 65k packets to all ports of a single IP within a few seconds, but NAT not punctured (DDOS protection / ?)
    • Not easy to debug a black box, but will try to experiment further
  • Relay protocol initial thoughts
    • API draft
      • –> REQUEST_CONNECT(PublicKey, Address, Challenge)
      • <– RESPONSE_CONNECT(ChallengeResponse || error)
      • –> SEND_PACKET(PublicKey, Payload)
      • <– RECEIVE_PACKET(PublicKey, Payload)
      • <– DISCONNECTED(PublicKey)
    • Notes:
      • Relayed packets should be encrypted
      • Imprecise bandwidth accounting – some packet loss is expected due to unreliable UDP and peer churn => tolerate e.g. 90% accuracy
      • Economics of bandwidth accounting – who should pay for relaying (sender/receiver/both?)
      • Payout: every X MB, the relay requests a payout and drops the connection if counter-signed block not received within a timeout
      • The relay server prefers peers that have the highest MB balance/refuses peers with a negative balance
      • Reuse tribler_bandwidth block type for compatibility, or use a custom block type (relay_bandwidth)?
  • Writing progress: BLE background + communication architecture in .tex

@synctext
Copy link
Member Author

synctext commented Apr 22, 2020

sprint progress, latest .PDF thesis:

  • thesis focus: scientific problem of freedom of communication, not bluetooth pairing and connectivity
  • if thesis problem description does not include "violation of end-to-end principle", "social welfare" it's not yet scientific enough; engineering-only. Also: https://tools.ietf.org/html/rfc3724#section-3
  • 65k congestion, buffer overflow check, dtermine the maximum rate with a throw-away SIM.
  • ask Google: https://www.google.com/search?q=standard+proxy+protocol+udp
  • "we asked 17 students to build upon our framework and create non-trivial dApps. The results are presented in this chapter. All students managed to produce operational code, validating the understandability and usability of our framework" etc etc
  • 2 week prototype - proxy subproject
  • Problem Description scientific storyline 🧑‍🔬
   == RFC 3724 ==
   One of the most common examples of network elements
   interposing between end hosts are those dedicated to security:
   firewalls, VPN tunnel endpoints, certificate servers, etc.  These
   intermediaries are designed to protect the network from unimpeded
   attack or to allow two end nodes whose users may have no inherent
   reason to trust each other to achieve some level of authentication.
  • The first decades of The Internet, trust was achieved because participants could largely be trusted.
  • For the last 15 years the "end-to-end principle" is broken by security measures to keep us safe: firewalls, VPN tunnel endpoints, NAT boxes, etc.
  • privacy-respecting identity, pub/priv key infrastructure, global reputation and trust systems critically depend on the end-to-end principle (freedom of communication and freedom of computation).
  • We define the trust-paradox as follows: the security measures that try to keep us safe (firewalls, VPN tunnel endpoints, NAT boxes, etc.) are also preventing us from making the Internet truly trustworthy.

@synctext synctext changed the title Universal connectivity using imperfect hardware Universal communication using imperfect hardware Apr 28, 2020
@synctext
Copy link
Member Author

synctext commented May 1, 2020

Title proposal: "Reliable Trustworthy Communication using faulty hardware", is more accurate and catchy. The harsh term fault is justified because IETF Internet Standards are violated by network operators. Trust-paradox is also linked in title now. Alternative to "Universal communication using imperfect hardware".
Question: what scientific problem did the superapp solve?

@MattSkala
Copy link
Contributor

MattSkala commented May 13, 2020

  • rewritten intro and problem description
  • research question? "Can we devise a protocol facilitating trustworthy device-to-device communication between smartphones under challenging network conditions without
    using a central server?"
  • experimentation with multihole UDP punching
    • send 65k packets/10 s – successfully punctured Vodafone NAT from KPN and home broadband, but probably reached session limit on T-Mobile network
    • 1k packets/30 s is the fastest safe rate for T-Mobile (=> usually successful puncture within 30 minutes)
  • the current experimental setup

image

@synctext
Copy link
Member Author

synctext commented May 13, 2020

👏 With this milestone it becomes realistic to dream of adding address book functionality. Thus yielding a zero-server secure social network. Even your thesis title could reflect this. But your framework can also be extended to a MusicDAO, zero-server AI, and is thus a generic Big Tech alternative library.

The rate limiter forces you to use a certain UDP puncture method. Either detect using the behavioral pattern (switches in external IPv4) or hard-code the IPv4 blocks like AS33915 Vodafone Libertel B.V.. Advertise a signed block of your current NAT behavior and puncture effectiveness?

Final thesis sprints:

  • TOFU-based social network. Please list that wisdom of Keybase in your thesis if fits. User can send around 🦄 messages to eachother and Magnet:https://
  • as listed before: ' "crazy NAT puncturing" focus; later do relaying as last resort (then focus afterwards on bandwidth tokens) '
  • Used bandwidth accounting in Trustchain?
  • Proxies for non-punctured connections?
  • Combining above: DHT-lookup --- which friend is now online and we can chat with? (with the reliability of a consumer product! this library always works)

Thesis remarks:

  • shameless ambition: this thesis provides the building block for replacing any Big Tech service. Google, Amazon, and Facebook full features can all be implemented on top of our library, albeit with possibly at least as much effort as these Big Techs have spend.
  • Good research question chapter, will only require minor polish
  • introduction material and tutorial on IETF should be replaced with the words the Internet is broken.
The Internet standards are
developed by the working groups of Internet Engineering Task Force (IETF), a nonprofit open standards organization composed of volunteers. Its evolution is based on
a rough consensus about technical proposals, and on running code. While there are
many conflicting opinions on its architecture, the general consensus is that the main
goal of the Internet is to provide global connectivity by the Internet Protocol (IP)
  • more dramatic chapter 3, no boring engineering stuff, replace entire sections with [15,17,18]. More like: Broken Internet technologies.
  • Sound sciency: system architecture and protocol design

@synctext
Copy link
Member Author

Problem Description: The Internet is under continuous attack
(Suggestion of alternative storyline)

Many Internet protocols, security companies, and security models have been compromised. Within this thesis we address a big open research question, can we make non-trivial software systems that are safe?

We identify three classes of attacks on The Internet: attacks on the protocol level, compromised central companies, and broken ecosystems.
Many protocols such as email are inherently unsafe to use. However, we still rely daily on such fundamentally broken Internet protocols.
Companies that are hires to secure other companies and banks have themselves been hacked. For instance, RSA using the SecureID two-factor authentication product (40 million customers) was infiltrated with a simple social engineering trick and a zero-day vulnerability in Adobe Flash.
The digital sectificates we use to keep ourselves safe are compromised. This brings into question the integrity of the digital-certificate model itself. Can we rely on trusted third parties to be trustworthy? We have seen time and time again that companies are not incorruptible saints.
The entire ecosystem of computer security is not provides users with safe computers. By not holding software vendors accountable for security problem they lack the direct incentive to address security. The security of endpoints is poor and financial damage done to users is never compensated. Software vulnerabilities in various machines has reached an significant level. To illustrate this point, we turn to botnets, which have automated the hacking of consumer and company computers. A study of Torpig explains that hijacked computers form a botnet, which itself may get hijacked. This is known as a botnet takeover.

These problems lead us to the following detailed problem description. Can we remove the reliance on any outside party, protect against malware, use open source software with proven software correctness, offer perfect forward secrecy for our communication, and protect against "Strong Existential Unforgeability under Chosen Message Attack"? Within this thesis we focus on the secure communication part within the grand question of how to make the Internet safe again. Other researchers within my research lab designed a system for trustworthy code module execution.

@synctext
Copy link
Member Author

synctext commented May 26, 2020

Latest .APK
Replacing Big Tech storyline inspiration
"3.4. P2P Communication Libraries" should be 3.1 and mention scuttlebutt.nz + briarproject.org
2004 related work, "DHT Routing Using Social Links"
TUDelft related work 100 Million DHT replies
Sub-second DHT paper

@synctext
Copy link
Member Author

synctext commented May 27, 2020

The scientific problem we focus on within this thesis is how can we bootstrap an ecosystem which will make The Internet safe and fair. The thesis approach to this ideological-driven scientific problem is to remove any central point from the ecosystem which can potentially be compromised or corrupted to act against the user interest.

We propose a zero-server architecture which removes structural bottlenecks for security, privacy, fairness and governance. This thesis provides irrefutable proof that our zero-server architecture is capable of solving wicked problems such as online voting, shared ownership of money, decentralised machine learning, bounty-based software engineering, fully automated decentralised marketplaces, and replace media industry intermediaries with code. We experimentally validated our zero-server architecture with software developers at master-level. A student development team with 17 members successfully created various non-trivial extensions to our foundational zero-server architecture within 10 weeks.

Our zero-server architecture offers various networking primitives. These serve as the basic building blocks for creating full fledged alternatives for the services of "trusted" third parties or Big Tech companies. They are:

  1. Identity layer with public key of your friends or businesses you interact with
  2. Secure end-to-end encrypted communication bypassing NAT boxes and other imperfect hardware
  3. Always connected to your friend using social-based zero-server overlay with automated Internet address discovery based on a UDP-puncturing capable DHT
  4. Secure messaging of any message size
  5. Distributed bookkeeping with tamper-resilience using Trustchain

This thesis builds upon the recent breakthrough of the superapp paradigm. We focus on the latest development in software engineering for smartphones because this is currently the dominant method for Internet access for a large part of the worlds population. In a very short period 1 million "mini-apps" have been developed for the popular Wechat ecosystem using the Superapp paradigm. [1, 2,3, 4, 5].

Our Superapp based on the novel zero-server architecture has several desirable properties. We believe that this thesis proves the first proof-of-principle of a disruptive open innovation ecosystem which is resilient against attacks by disrupted or disintermediated corporate entities. Especially legal intimidation, lawyer-based attacks and creation of chilling effects. Within our 10 week experiment several mini-apps where developed successfully.

@devos50
Copy link
Contributor

devos50 commented Jun 8, 2020

Related work for a potential storyline: An Economic Theory of Infrastructure and Commons Management.

This work contains a discussion on the management of the digital commons. Section IV: "At the heart of this debate is whether the Internet should retain its end-to-end architecture and continue to be managed in an openly accessible manner."

@synctext
Copy link
Member Author

synctext commented Jun 8, 2020

LATEST .pdf of thesis

  • related work, now you also are superior (in terms of architecture) to mastodon federated social network.
  • Chapter 2 still 'modest' problem description, ToDo, the universal platform ambition..
  • latest .APK with animated GIFs 😮
  • show feeds from friends, MvP: social network claim within thesis?
  • Do a thesis writing sprint: put screenshots in implementation chapter and claim strong results (split designs or one giant chapter, DAO, communication lib, etc.).

@synctext
Copy link
Member Author

synctext commented Jun 9, 2020

Reclaiming The Internet for the social good

This master thesis empowers citizens to take back control of The Internet. More specifically, we present the self-organising technology stack to take back The Internet. Who owns The Internet? The Internet is essentially private property, with few exceptions. Google, Facebook, Amazon, Apple, Tencent, and others operate the central components of our daily digital lives. For instance, we require permission from Google and Apple to publish software for mobile devices. Their monopoly power means no other meaningful method exists to reach billions of smartphone users with newly created apps.

This thesis introduces a novel type of low-level network overlay and proof-of-principle zero-server social network. Our zero-server architecture offers various networking primitives. These serve as the basic building blocks for creating full fledged alternatives for the services of "trusted" third parties or Big Tech companies. These primitive increase in strength and cover identity management, communication, code execution and social software engineering.

Within the context of this thesis we conducted an experiment with computer science students. A student development team at master-level with 17 members successfully created various non-trivial extensions to our zero-server architecture within 10 weeks. With their efforts the "superapp" presented within this thesis provides operational features such as online voting, shared ownership of money, fully automated decentralised marketplaces, and a Spotify alternative.

The full list of Superapp features, at an early proof-of-principle stage of software development:

  1. Secure address book. Obtain the public key of your friends with protection against man-in-the-middle attacks. This forms the identity layer with public key of your friends or businesses you interact with
  2. Universal communication using imperfect hardware. Secure end-to-end encrypted communication bypassing NAT boxes and other imperfect hardware. Connect to any address book entry in a secure manner. Secure messaging of any message size.
  3. Robust social fabric. Discover the current Internet address of your friends and connect to them. When your friends or trusted businesses come online they will form an end-to-end encrypted connection with you. Address churn by creating a social-based zero-server overlay with automated Internet address discovery, based on a UDP-puncturing DHT.
  4. Community formation. Everybody can freely start a community which can grow to any size. This is the core of our Decentralized Autonomous Organization. Key features are discovery of DAO communities and joining them.
  5. Community accounting. For each address book entry we track their contributions to the common good. Distributed bookkeeping with tamper-resilience using Trustchain
  6. Democratic governance. Community members can propose votes with proposition for financial transactions.
  7. Shared ownership of money. These communities of any size can democratically control any amount of money. By utilising Bitcoin Multisig we provide a community of any size the democratic control for any amount of money.
  8. Distributed Apps. Freely executing code forms the beating heart of the DAO. Improvement of smart contract execution model, specifically optimised for smartphones usage with runtime code injection and bypass the Google security model.
  9. Distributed markets. Marketplace with bid/ask orderbook which is distributed.
  10. Bounty-based software engineering. Any developer can seed a distributed app in Bittorrent and a DAO can vote on payout of bounty.
  11. Decentralised machine learning. Without any central point of control or performance bottleneck perform multi-dimensional curve fitting.
  12. Distributed music industry. Support artists by making them self-published, self-promoted, and self-sufficient in general. Replace the music industry offerings with free software. Grown into a full master thesis aiming to remove all proprietary infrastructure between artists and ears of listeners with cost-free ecosystem.

@synctext
Copy link
Member Author

synctext commented Jun 17, 2020

The superapp is new paradigm in software engineering. Users have an app with numerous possibilities each of them integrated and deep understanding of your social graph and who you trust. No leakage of this vital information to others or central servers. It also serves as a social coding platform where each module contribution gets global visibility without the burden of marketing and (fake) app review hunting.

Progress: social network is operational in Beta with friend feeds 🥇 Tribler/trustchain-superapp#41

Thesis writing first now, online profiles with photos are secondary. ToDo: 1) scientific paper by 31 Aug in 6-pages of your work 2) Google Play store listing of superapp 3) a few key screenshots with epic something..

Engineering first principle..: "5.1. Project Structure" and only then the scientific key insight of "5.2. System Architecture". Likewise "5.7. Library Usage", "5.7 creating a community"; we aim to germinate a community through software re-usage and defining an IETF Internet Standard around this work. Wild idea for results section, buy positive reviews and document results for a stand-alone app; stay legal. ToDo: graphs with experimental results in key "performance evaluation and experiments" chapter. Response time with 1000 friends in your address book, etc.

@synctext
Copy link
Member Author

synctext commented Jun 24, 2020

Supertrust: fixing the Facebook monopoly

Roughly 2 billion humans interact with Facebook every day, it has a 75% market share. Facebook has chosen not to compete on the merits of its products and services and instead has misled, deceived, and exploited consumers and publishers. Evidence discussed in an analysis paper shows that for 10 years Facebook has avoided competition in the social network market. Professor Fiona M. Scott Morton from Yale University states that: "Facebook has engaged in a long-term, integrated, anticompetitive strategy of half-truths about its privacy policies, exclusionary API manipulation, and anticompetitive acquisitions of nascent competitors that led to its current dominance of a market in which it now wields significant power over consumers, advertisers, and publishers".
This master thesis is dedicated to the recommendation made within this study by economists of anti-competitive conduct: "remedial measures might need to include mandatory interoperability among competing platforms and complements". We successfully created the architecture plus implementation of our foundational layer for interoperability and data portability to all social network platforms. Our work consists of 3 parts: our zero-server architecture, fully operational network primitives, and extension of features using mini-apps.

We propose a zero-server architecture for social networking which removes the need for central control by Facebook. This thesis provides irrefutable proof that our zero-server architecture is capable of creating the first interoperable product between various social network platforms. No central element exists which can form a performance bottleneck or point-of-failure. Our zero-server architecture offers various networking primitives. These serve as the basic building blocks for creating full fledged Facebook alternative with full interoperability and data portability. We have not created a full alternative to Facebook within the context of a single master thesis. However, we provide the basis for it and claim that our infrastructure will scale to usage by the 2 billion existing Facebook users without fundamental alterations. Our Internet-deployed network primitives are:

  1. Identity layer with public key of your friends
  2. Secure end-to-end encrypted communication bypassing NAT boxes and other imperfect hardware
  3. Always connected to your friend using social-based zero-server overlay with automated Internet address discovery based on a UDP-puncturing capable DHT
  4. Secure messaging of any message size
  5. Distributed bookkeeping with tamper-resilience using Trustchain to prevent spam and abuse

Our zero-server architecture is also meticulously designed to offer another unique property: resilience against legal attacks. Legal action may be expected when a monopoly is attacked. We believe that this thesis proves the first proof-of-principle of a disruptive open innovation ecosystem which is resilient against attacks by disrupted or disintermediated corporate entities. Possible legal attacks includes: legal intimidation, lawyer-based attacks and creation of chilling effects. Our unique approach to governance gives ownership to both nobody and everybody. By creating an academically pure form of self-organisation we create a leaderless infrastructure in which even the original creators have no special powers. It is difficult to block, disrupt or shutdown our zero-server approach with either legal or technological means when used by citizens at scale. On top of our networking primitives we facilitate an open innovation ecosystem. Every user can extend the functionality of the platform freely, called "mini-apps". No gatekeepers exists to moderate mini-apps, this relies on self-moderation by the community. We conducted an experiment around the mini-apps to determine how easy they are to develop. A student development team with 17 members successfully created various mini-apps with our zero-server architecture. Developed mini-apps within a 10-week period include: group creation and shared ownership of money, secure online quorum-based voting, decentralised machine learning, and sharing of music recording.

@MattSkala
Copy link
Contributor

  • latest pdf report: https://github.com/MattSkala/msc-thesis/raw/master/report/report.pdf
  • updated problem description and research question - shifted focus from connectivity to decentralized stack for providing alternatives to big tech services
  • thesis now consists of 2 parts:
    • zero-server architecture and networking primitives
    • decentralized super app alternative with proof-of-principle decentralized social network
  • screenshots of the super app in chapter 6

@synctext
Copy link
Member Author

synctext commented Jul 8, 2020

  • "3.3. NAT Traversal" techniques
  • "In this chapter, we design a protocol facilitating peer to peer communication between any devices.". Opening line of chapter needs more "Facebook alternative" audacious ambition.
  • Make also audacious ambitious, not boring: implementation chapter opening lines "In this section, we describe the system architecture and implementation of the P2P communication library. One of our main goals is to create an implementation that would be compatible with the majority of mobile devices."
  • Chapter 6: more like, we present our implementation which provides an alternative for the app market of the two monopoly services within this market: the Google Play Store and Apple marketplace. It provides future-proof upgrades of functionality and lets ecosystems grow and expand over time with an stable integrated foundational layer.
  • Chapter 7: need exciting evaluation stuff, like end-to-end latency measurements and breakdown for chat, TFTP and stuff. DHT lookups in the wild?

@MattSkala
Copy link
Contributor

  • more broad title proposal: "Towards Decentralized Mobile Services: Technology stack for decentralized mobile applications"
  • last week focused on experiments, plots and evaluation:
    • bootstrap performance – duration to get from 0 to 30 peers
    • messaging complexity and latency when growing from 20 to 500 peers
    • TFTP binary transfer rate
    • power efficiency

@synctext
Copy link
Member Author

synctext commented Jul 20, 2020

  • Less NAT engineering, more science of Big Tech alternatives
  • Thesis naming
    "Decentralized Mobile Services: towards a Technology stack"
    "Superapp: towards a decentralised technology stack"
    "Superapp: towards a Big Tech alternative"
  • 7.3. Message Complexity and Latency. Scientific: "7.3 social graph scalability experiment"
    You are using the live network for accurate real-world performance quantification
    Not sold at all!
    Kbps is preferred.

@synctext
Copy link
Member Author

synctext commented Aug 24, 2020

Near final thesis .PDF of 60 pages

In general, this thesis does not claim any of the scientific breakthroughs that are achieved. Its really solid work and important step towards Big Tech alternatives.

Remarks:

  • "2.3. End-to-End Principle Challenged"
    Add a single opening sentence which spoilt the whole section and introduces why this 3rd item is introduced within the problem description. Like: At the business-level Big Tech monopolies have created a monoculture, at the network level problems are also created due to the architectural violations of "middleboxes". We can no longer freely communicate with every other Internet users, for our own protection.
  • Numerous missing citations. Bramble QR Code Protocol (BQP) [? ].
  • 3.1.4 : a single subsection has a "Conclusion", other not.
  • 3.2. Background on State of the Art
    No connection between Big Tech dependency and critical infrastructure dependency on ISPs with their introduction of middleboxes. Add 1-2 lines in the opening on that.
  • 3.3. NAT Traversal
    Instead of the detailed technology, shift intro storyline to "permissionless communication". Bypass those middleboxes.
  • 3.4 repeating: communication without infrastructure..
  • Chapter 4, more self-promotion: Any alternative to Big Tech requires a permissionless communication protocol. In this chapter we explain our design.
  • Design Chapter: "Generic Attribute Profile (GATT)" too much tutorial detail.
  • Chapter 5: We successfully implemented a generic library for Big Tech alternatives, facilitating generic and secure P2P communication.
  • "This idea is highly related to the previous research on FBase" [Citation]
  • 7.4. Binary Transfer Rate: no motivation, communication any message of any size to any Internet user
    : Raw datapoints: Figure 7.5: The relation between the number of connected peers on the battery usage
  • 8.1. Future Work; remove this heading simply without text changes or add an 8.2 section
  • Scientific architecture: how ideas fit together and how all engineering pieces fit (missing chapter).

@ElBrujo
Copy link

ElBrujo commented Sep 3, 2020

I'm pretty impressed, I'll love to follow your steps and checked all the code and notations.

This thesis has been deployed to make successful technology, I think, could implement a low-cost repeater, I figured that like a patrol of drones making surveillance in a way of "machine learning distributed network" like a microsatellite, I think we will put some low-cost hardware with comunicatión by laser to extend a network across larger distances... with all the respect what all of the people in this project deserves, I want to implement some of these big tech stuff you are making for real. Thanks and now you are making a decade of investment and deployment a successful protocol, library, and framework, congrats!

@synctext
Copy link
Member Author

synctext commented Oct 26, 2020

Technology Stack for Decentralized Mobile Services

THESIS FINISHED. Direct download of raw .PDF with 59 pages of text

btw Seems the idea itself of P2P apps got already introduced by Stanford in 2012, but never realised. Until Matt did it!

Musubi attempts to create a social mobile INTERNET where users can have a rich and carefree social
sharing experience with any group of friends, with the freedom to choose their own cloud storage provider
and apps on their mobile devices. This open architecture can disrupt the lock-in seen in existing social
INTRANETS which imposes rigid control over both the users and the developers.

https://mobisocial.stanford.edu/papers/musubi_disintermediated.pdf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants