• These systems typically still have single points of failure. The content creator and provider still needs to host a server which contains a list of caches that are currently holding the content. If the content provider's server goes down, users can no longer access the content. This makes these network particularly vulnerable to censorship.
• Secondly these systems introduce new kinds of financial attacks on the network. Content clients and caches can collude to receive payment even when content has not been transmitted. Though solutions exist for mitigating these so-called cache accounting attacks, the overhead they introduce effectively renders real-time content delivery impossible [3, 4].

Figure 1: Different CDN architectures

• Content routing is a core component of the Myel CDN. It directly impacts how quickly content is delivered to end users. Here we define two key aspects:

• How content is dispatched to cache providers

  • When Myel nodes add content to the network, the content is dispatched to specific groups of cache provider nodes. This new content is advertised to the caches with a gossip messaging protocol. Subnetworks of providers are grouped around gossip topics and clients target relevant caches by publishing to those topics. It is then the cache providers' decision to accept the request and retrieve the content from the client to serve it for future requests. Note that subnetworks are primarily created around geographic regions, but can also be around communities who share interest for similar content.

    • Figure 2: Clients target relevant caches by publishing new content to a relevant gossip topic. This content is then hosted by subnetworks of providers associated with that topic.

    • This system has been implemented and is currently being tested.

    •  // Content is dispatched with a simple call
        err := exchange.Dispatch(c cid.Cid)

    • Further, we introduce an automated mechanism for providers to select which content they should serve. This mechanism allows for the network to regulate the lifecycle of content on the network and grow the number of providers hosting a piece of content to meet increases in demand. Underpinning this are two principal mechanisms:
    • On top of the initial cache dispatch, providers can ask each other for their retrieval logs in order to refresh their supply with popular content.
    • This background job routine can also be enhanced with a probabilistic model to determine the best provider-content pairing. Local nodes continuously keep track of the content they serve and the subsequent rewards received for that content. A lightweight probabilistic model, such as logistic regression or a random forest, can then infer what kinds of content (for example static pages versus videos) are most profitable for the node to host.

      • This still needs to be proven, our thesis is that it will greatly optimize the performance of the network.

• How content is discovered by clients

  • Fast and efficient strategies for content discovery are critical to the functioning of the Myel network. This is the process by which a client is able to find a cache that is hosting the content they are requesting. Unlike a traditional CDN or a collaborative cache CDN, which use centralized indices of content-cache pairs to route content to a client, the Myel network needs a content discovery system by which a client can quickly search the network for the they are looking for, without relying on a single centralized index.
  • Traditionally, a HTTP request connects to a DNS which then returns the address of a CDN point of presence (POP), if the POP has the content cached it returns immediately or asks the origin server for it, caches it and finally returns it. This is very simple and performant but assumes we can trust the DNS provider, the CDN provider and the origin server. If any part of the system fails or the link is corrupted users have no alternatives to access their content.
  • Myel uses interplanetary linked data structures (IPLD) and its immutable content identifiers (CID) to verify the integrity of content and declare encoding and hashing functions explicitly. In this data structure, content is chunked and split into different sized blocks forming a directed acyclic graphs (DAG). IPLD makes the system interoperable and resilient whilst remaining flexible such that it can be used for any kind of data model.
  • Developers can use the naming system of their choice (e.g. IPNS, ENS, etc.) to point to a CID representation of the content. Any time the content is mutated, the naming record should be updated with a new CID. In addition, IPLD selector specs are available to declare which parts of the content graph to query. This can be represented via a simple path notation, for instance:
  • <CID>/images/Frog.jpeg
  • or a more complex serialized declaration for traversing a graph and returning specific nodes. By default, queries for a root CID fetch the entire graph.
  • Myel offers multiple mechanisms for content discovery based on the use case. The first, gossip-based, is similar to Gnutella and fully decentralized though may have a higher latency, the second Hub-based offers more performance with a compromise on decentralization.
    • Gossip System: Content discovery begins by publishing a gossip message containing a CID and selector to one of the provider subnetworks. The gossip message is propagated across the network of providers who check their supply and decide if they want to reply with an offer or simply ignore the message. To send the response, each peer recursively forwards the message back to the publisher. This adds more privacy as it becomes harder to trace a query back to a client as the network grows. The offer contains terms under which a provider is willing to accept a data pull request.
      • Depending on the client preference (i.e. faster or cheaper transfer), an offer is selected and the client creates a new deal and opens a data pull request with the provider. The provider validates the deal and if it is compatible with the offer will initiate the payment protocol.

  •   // Start by creating a new session for a root cid
      session := exchange.Session(context, cid)
      // Query the network for the best offer
      session.QueryGossip(context)
      // Start the transfer when we have a satisfying offer
      session.StartTransfer(context)
      // Wait for the transfer to complete
      error = <- session.Done():
      // if err == nil the transfer was successful and content is in our blockstore

  • Hub System: Another system for discovery on the Myel network, in the vein of Gnutella2 [5], is to create a special class of cache called Hub nodes, which index content-caches pairing in their local area. Whenever a cache receives new content, they inform their nearest hub to update its index. Whenever a client wants to access content they then query their nearest Hub for the cache address that hosts the desired content. Hubs can also cross-communicate so as to sync indices over a larger group of caches.
    • By having multiple hubs, rather than a single centralized index, the network is more resilient to downtime and censorship. Nevertheless this system increases the complexity of the network, and, relative to other possible solutions, still requires safeguards to prevent a malicious actor from using the network for denial-of-service attacks -- as an attacker only needs to attack a select few hubs rather than the entire network. This vulnerability decreases as more hubs are present in the network, and as more inter-hub communications are formed.

  • Figure 4: Hubs maintain an index of cache-content pairs. In green we show communications where a cache informs a hub of the content they are hosting. In blue we show communications where a hub returns the cache address that holds the content a client is looking for. The client can then query that cache for the content. Hubs can also communicate to sync their indices such that each hub covers a larger group of caches.

• In the Myel CDN implementation, payments are executed directly between cache providers and clients retrieving the content. To enable direct P2P payments, Myel uses existing crypto payment protocols.

• Currently, the data transfer protocol is based on the Filecoin retrieval protocol and designed to operate in a trustless environment in which a provider may attempt to send bad content to a client and still receive payments. To that end, payments are divided into multiple intervals to let the client validate that the correct content is being transferred. If one of the parties decides to interrupt the transfer, the provider will still receive payments for some of the transferred blocks. As more blocks are successfully validated by the client, the payment interval can increase to include more blocks per payment and increase the speed of transfer.

• Figure 3: Data transfers are batched into n data chunks. After the transfer of each chunk of data, the cache receives a fraction 1/n of the total payment.

• Centralized CDNs can currently handle up to 800 billion requests per day [8]. Each request on the Myel network generates n payment transaction messages. At the scale of centralized CDNs the network would need to handle n x 800 billion messages a day. To avoid bottlenecks and make the system inherently scalable, most of those messages are generated off chain and accounted for in state channels that can be resolved on chain asynchronously. Those payment channels are created on the Filecoin blockchain, though other blockchains could be used to issue payments.

• As mentioned previously, when content is transferred, payment happens reccuringly at an agreed upon payment interval. At this interval providers can pause transfers to request payment from the client, with the steps as follows:

  • 1. The client signs and sends a voucher message with the requested amount.
  • 2. The provider then stores those messages and can aggregate all the payments into a single blockchain transaction to redeem the payment.
  • 3. Filecoin provides built in smart contracts to process payment vouchers and adds a 12h settlement period during which any of the two parties involved can submit valid vouchers. Once the settlement period is over, the provider can collect the resulting amount on their address.

• Myel Actor

  We use a custom actor (also known as smart contract) on the Filecoin blockchain as a proxy between client and providers. As a result, clients only need a single payment channel with the Myel actor in order to pay for all their transfers. A the time of this writing, Filecoin does not support custom actors but it is in the work and we are able to prototype the features of the Myel contract to validate this architecture. We will be adding more details on this soon.

These consumption numbers are not reflective of traffic or pricing on the Myel network and we will update these numbers as we gather more data on consumption habits and optimal pricing levels.