ILP Torrent - The technical deep dive

ILP Torrent – The technical deep dive

October 26, 2020

I recently introduced ILP Torrent, an implementation of a torrent protocol based on WebTorrent that includes verification of payments from downloaders to content creators. What was not included in that post is a more in-depth explanation of how a torrent protocol in general works and what changes I made to the WebTorrent code base in order to allow for payments. Hopefully, this is going to make up for that.

The BitTorrent protocol – The short and simple version

The BitTorrent protocol is a peer-to-peer network for file sharing of all sorts. Some people use it to share music and movies (legally and illegally), companies use it to deploy new code on their servers, and open-source projects use it to fastly deliver files among the community. BitTorrent was not the first peer-to-peer file sharing protocol, there were Napster and Gnutella before it. However, it is the most efficient one because instead of downloading an entire file from one peer (like in Napster and Gnutella), files are split into parts and a torrent client downloads these parts from multiple peers.

Image from Wikimedia Commons and altered to be in English.

Let’s digest the above schema. On the right hand side we have a Web Server that provides .torrent files. For example, Ubuntu.com makes their popular Linux operating system available to download via Bittorrent and publishes the .torrent files accordingly. The client on the bottom downloads that .torrent file from the Web Server. Such a .torrent file includes the following information (source: Wikipedia):

- announce – the URL of the tracker(s)

- info:

- length – size of the file in bytes

- name – suggested filename where the file is to be saved

- piece length – number of bytes per piece. This is commonly 28 KiB = 256 KiB = 262,144 B.

- pieces – a hash list, i.e., a concatenation of each piece's SHA-1 hash. As SHA-1 returns a 160-bit hash, pieces will be a string whose length is a multiple of 20 bytes. If the torrent contains multiple files, the pieces are formed by concatenating the files in the order they appear in the files dictionary (i.e. all pieces in the torrent are the full piece length except for the last piece, which may be shorter).

A Tracker (on the top) is another server in the system that keeps track of who is currently making which piece of a file available, i.e. who is part of a Swarm. There are two types of participants in a Swarm, Seeders and Leechers. Seeders have the entire file, i.e. all the parts, available and may be the original creators of that file or have completed the download. Leechers are in the process of downloading a file and only have some parts available for others to download from them. The network participant on the bottom, the one that requested the .torrent file from the Web Server, is a Leecher because they only have 37% of all parts downloaded.

Instead of obtaining a .torrent file, a Leecher may also use a Magnet URI to identify a Tracker and to request a peer list from it. A Magnet URI is at least comprised of

- ih: info hash

- tr: tracker

and has the following format:

magnet:?xt=urn:btih:b958ace21246c1bda46d332db918f750502b555b&dn=protea.jpg&tr=wss%3A%2F%2Filp-torrent.sabinebertram.com

It includes an endpoint to the Tracker and a hash to identify the file. Additionally, it may include the file name (dn), such that torrent clients can display it during the download process.

Enabling payments in a torrent protocol

We want the creator of a file to be rewarded for their work. Hence, we need to add a license to the .torrent file to include all of the following information:

- announce

- info

- license:

- paymentPointer – proxy payment pointer hosted at a STREAM receipt verifier. This payment pointer proxies SPSP queries to the creator’s actual payment pointer hosted by the wallet, enabling STREAM receipts. (*)

- verifier – STREAM receipt verifier endpoint

- amount – price of the file, e.g. “100”

- assetCode – asset code the amount is denoted in, e.g. “USD”

- assetScale – asset scale of the amount. To avoid expressing decimal numbers all amounts are expressed as integers with an asset scale applied.e.g. “US dollars are commonly expressed with an asset scale of 2”, meaning that an amount of “100” corresponds to 1 USD (100 divided by 10 to the power of the scale: 2).

(*) A blog post on STREAM receipts and the STREAM receipt verifier is coming soon.

The Tracker will use the verifier endpoint to spend the file price as specified in amount, assetCode, and assetScale, before communicating a peer list. Currently, the whole price must be paid before the peer list is shared. In the future, the tracker could be upgraded to allow users to pay continuously as they request updated peer lists for downloading. This makes especially sense for downloading larger files like movies, which would probably also be more expensive than an image.

Payment pointer and verifier are also included in the Magnet URI, together with the usual information like info hash, file name, and tracker endpoint.

This allows the torrent client, in our case the browser, to add a Web Monetization meta tag to the website and to credit STREAM receipts.

The seeding and leeching flow

Let’s go into detail about how seeding and leeching works in ILP Torrent.

(1) Alice the Seeder wants to seed a file. She specifies the license as outlined in the previous section and submits that to her ILP Torrent Client (Alice’s Client).

(2) Alice’s client generates the .torrent file, also including the license. Additionally, it generates the magnet URI.

(3) Alice’s client announces the new file to the tracker, who remembers that Alice is a peer. It also parses the .torrent file and learns about the license.

(4) Finally, Alice’s client shares the magnet URI with Alice.

(5) Alice communicates the magnet URI to Bob the Leecher, who wants to download the file.

(6) Bob enters the magnet URI into his Client (Bob’s client)

(7) Bob’s client decodes the magnet URI and learns about the info hash, file name, tracker endpoint, payment pointer, and verifier endpoint.

(8) Using the payment pointer, Bob’s client starts streaming micropayments to Alice’s wallet.

(9) If the payments are successful, Alice’s wallet issues receipts to Bob’s client.

(10) Bob’s client submits the received receipts to the verifier, using a balance id (e.g. the Web Monetization request id).

(11) The verifier credits the receipts using the balance id.

(12) Using the balance id, Bob’s client now tries to retrieve a list of peers from the Tracker.

(13) The Tracker spends the receipts against the verifier, i.e. verifies them, and if that is successful, i.e. the receipts are valid and the balance is sufficient to pay the specified amount in the license,

(14) it responds to Bob’s client with a list of peers.

(15) Now, Bob’s client connects to Alice’s client and downloads the file part by part.

The implementation: Tweaking WebTorrent

Since I wanted to use Web Monetization micropayments to pay for the torrent license fees, WebTorrent was the obvious choice for a baseline. In contrast to most torrent clients like μtorrent or the BitTorrent Client, WebTorrent is a streaming torrent client for the browser (and node.js) that uses WebRTC for peer to peer communication. Hence Web Monetization can be used in conjunction with it.

As already highlighted in my non-technical blog post, WebTorrent comes with a bunch of modules of which I had to change:

- magnet-uri

- create-torrent

- parse-torrent

- bittorrent-tracker

- torrent-discovery

- and webtorrent itself

The dependency structure is depicted below:

Let me tell you about the tweaks I had to make.

magnet-uri

This package encodes and decodes magnet URIs. In order to work for ILP Torrent, I added functionality to also encode and decode the payment pointer and the verifier endpoint.

⇒ ilp-magnet-uri

create-torrent

This package creates .torrent files. It now includes functionality to add a license to the .torrent file.

⇒ ilp-create-uri

parse-torrent

This package parses .torrent files. Additions were made to also allow for parsing the license included in the .torrent file. It uses ilp-magnet-uri.

⇒ ilp-parse-uri

bittorrent-tracker

This package is 2-sided. It includes a client and a server where the client is used to connect to the server. On the client side, I changed extended WebSocketTracker to handle license information as well as the web monetization request id. I also disabled connection via HTTP and UDP since I don’t support these yet. Changes on the server side were more substantial. I extended the Swarm class to deal with receipt verification by spending against a designated endpoint at the verifier, using the web monetization request id. It retrieves the amount it should spend from the license. Only if this spend is successful, the server communicates the peer list.

⇒ ilp-bittorrent-tracker

torrent-discovery

The only change I made was to use ilp-bittorrent-tracker instead of bittorrent-tracker.

⇒ ilp-torrent-discovery

webtorrent

Finally, the webtorrent package uses ilp-create-torrent, ilp-parse-torrent, and ilp-torrent-discovery.

⇒ ilp-webtorrent

The code can be found here.

Improving usability: The STREAM Receipt Verifier Service

The first version of the demo of ILP Torrent, which was not announced publicly, required a content creator to have set up their own STREAM receipt verifier. Even though it is not hard, a virtual server is a prerequisite. Furthermore, to ensure security, it should be linked to a domain and TLS should be set up.

To reduce the burden and to make the demo easier to play with, I advanced the STREAM receipt verifier to allow for multiple payment pointers to be proxied. The receipt accounting, however, did not change and still only relies on a balance id, e.g. the Web Monetization request id.

In detail, I added an API that allows you to register and deregister your payment pointer. It creates a proxy payment pointer by hashing the original one and appending it to the verifier’s SPSP endpoint, e.g.

https://ilp-torrent.sabinebertram.com/verifier-spsp/9b73ec640f5a5e88182fcd13ad4e8417ae2f366ab3baaa60cc3e077ea0cb641b

When the Web Monetization provider (e.g. Coil extension) requests STREAM credentials using this proxy payment pointer, the Verifier Service uses the hash to look up the actual payment pointer and forwards the request to the user’s wallet while also sharing the STREAM receipt secret. The response is then also forwarded to the Web Monetization provider, who opens a regular STREAM connection.

The code is available here and the STREAM Receipts Verifier Service for the ILP demo can be found here.