Comprehensive BitTorrent Protocol Analysis: Peer Data Transmission and Handshakes Explained

0. Review

Previous articles:

• Introduction to the Bittorrent Protocol (Part 1) Metadata Files https://cloud.tencent.com/developer/article/2332701
• Introduction to the Bittorrent Protocol (Part 2) Tracker and Peers https://cloud.tencent.com/developer/article/2333043

Review of previous content:

• BitTorrent is a protocol for distributing files, utilizing metadata files encoded with bencode, performing SHA-1 hash comparisons on pieces, and introducing the metadata file structure
• Analysis of Tracker Get requests and examples
• Analysis of Peers communication

In the Peers communication analysis section, the detailed description of the actual transmission can be difficult to understand. This article will build upon the previous articles and provide a detailed analysis of the data transmission content between Peers through an example.

1. Data Transmission Example

Environment

1. Using Sockit tool for TCP data sending and receiving, Sockit is an open-source tool based on C++, more details: https://github.com/sinpolib/sokit or http://code.google.com/p/sokit/.
2. Using Transmission version 4.0.4 as the peer downloader. Pre-download, install, and configure, randomly select, and set the listening port. In this article, the port is 51045.

Prepare Test File and Metadata File

Generate a bin file of 256 KB using the following Python script

with open("test.bin", "wb") as file:
    for j in range(1024):
        for i in range(0, 256):
            file.write(i.to_bytes(1, byteorder='big'))

Compute its SHA1 value to complete the metadata file

{
    "announce": "http://10.0.150.202:6969/announce",
    "info": {
        "length": 262144,
        "name": "test.bin",
        "piecelength": 33554432,
        "pieces": b"7\xefwio\xc2U\xbfS\xb4\xcd\xd0\x14\xb2#go-\xc8\xb",
        "private": 1
    }
}

To prevent Transmission from reporting data on anonymous networks causing uncertainties in the test, a private flag, not previously mentioned, is added here. Transmission will recognize this flag and will not share this task through DHT, PeX, LSD, etc. network requests and transmissions. If there is an opportunity, further analysis may include this in future articles.

Alternatively, you can disable Transmission’s anonymous peer exchange connections directly in the settings.

Perform Bencode encoding on the info dictionary, compute SHA 1, and obtain the metadata information hash:

1ae5136ee599a6d67913d5ab6a44a4efdfa681e4

Import the metadata file into Transmission, verify and start downloading (seeding):

Transmission Import

Establish Connection and Handshake

The Tracker request was previously described and analyzed in detail; thus, we will not repeat it and move directly to the Peers data transmission content.

First, generate a Peer ID, which is a 20-byte random string. Many downloaders on the market use -identifier-random content to include their downloader’s identifier in the Peer ID, such as qBittorrent’s Peer ID starting with -QB{version number}-; you can choose any identifier for tests. This article uses -Qcloud-thisisrandom as the Peer ID.

Establish a TCP connection to the Transmission port using Sockit and send handshake information encoded in hexadecimal as follows:

13 42 69 74 54 6F 72 72 65 6E 74 20 70 72 6F 74 6F 63 6F 6C 00 00 00 00 00 00 00 00 1A E5 13 6E E5 99 A6 D6 79 13 D5 AB 6A 44 A4 EF DF A6 81 E4 2D 51 63 6C 6F 75 64 2D 74 68 69 73 69 73 72 61 6E 64 6F 6D

Details are as follows:

• 0x13 (19) protocol length, fixed to 19
• 42 69 74 54 6F 72 72 65 6E 74 20 70 72 6F 74 6F 63 6F 6C Bittorrent protocol string
• 00 00 00 00 00 00 00 00 Reserved bits, request does not use any extensions, all set to 0
• 1A E5 13 6E E5 99 A6 D6 79 13 D5 AB 6A 44 A4 EF DF A6 81 E4 Information hash, as previously mentioned
• 2D 51 63 6C 6F 75 64 2D 74 68 69 73 69 73 72 61 6E 64 6F 6D Peer ID, ASCII encoding for -Qcloud-thisisrandom

After sending, the handshake response received from Transmission:

13 42 69 74 54 6F 72 72 65 6E 74 20 70 72 6F 74 6F 63 6F 6C 00 00 00 00 00 10 00 04 1A E5 13 6E E5 99 A6 D6 79 13 D5 AB 6A 44 A4 EF DF A6 81 E4 2D 54 52 34 30 34 30 2D 68 70 66 63 6D 76 64 74 6F 79 38 6D

Handshake Information

In the diagram, before disconnecting, Transmission sent bitfield and unchock data, which will be analyzed later; first, examine the first 68 bytes of the handshake response received:

• 13 42 69 74 54 6F 72 72 65 6E 74 20 70 72 6F 74 6F 63 6F 6C Same as the sent part
• 00 00 00 00 00 10 00 04 Extension bits, Transmission announced extensions support but will not be utilized this session as they were not used in the handshake initiation
• 1A E5 13 6E E5 99 A6 D6 79 13 D5 AB 6A 44 A4 EF DF A6 81 E4 Information hash, consistent with the sent content
• 2D 51 63 6C 6F 75 64 2D 74 68 69 73 69 73 72 61 6E 64 6F 6D Peer ID provided by Transmission, this process ignores matching with the node list from the Tracker response, -TR4040-hpfcmvdtoy8m ASCII encoding, indicating that the Peer ID for this data from Transmission is -TR4040-hpfcmvdtoy8m

Bitfield, Unchock, Interested

After completing the handshake, both parties should exchange BitFields. Following the handshake, BitField data from Transmission is already received:

00 00 00 02 05 80

Once the handshake is complete, all integer data is encoded in 4-byte big-endian format. The number of data received refers to the message length, meaning 00 00 00 02 represents that the message length is 2 bytes, and the first byte of the message denotes the type flag 05, corresponding to Bitfield (see Introduction to Bittorrent Protocol (Part 2) Tracker and Peers), based on the file size and piece size in the information, it’s known, this content has only one piece. Therefore, the piece bitmask has only 1 bit, and the rest are padding. The received data 0x80, which is 1000 0000 in binary, indicates that Transmission informs having the first piece.

Note that some downloaders may intentionally set some positions to 0 during BitField announcement, then inform that they have this piece through Have messages.

Transmission also sent an Unchock message

Transmission Unchock Message

[00 00 00 01 01]

Similarly, the message length is 1 byte, type is Unchock, with no additional content required. Receipt of this message indicates that Transmission has unlocked uploading access. At this point, inspecting the resource properties in Transmission shows:

Resource Properties in Transmission

Now, send an Interested message to express interest in the pieces of data Transmission holds. The sent Interested message is just one byte indicating the type as follows:

[00 00 00 01 02]

When one side expresses interest and the other is unchocked, the former can request data from the latter, which will then transmit the data. Correspondingly, the status in Transmission is as follows:

Able to Transfer Data

Request and Piece

A BitTorrent piece can be too large, not suitable to send in its entirety at once. Therefore, during transmission, it is split again for transfer, requiring the offset and the size for each request piece of data.

Upon being ready to upload, data transmission requests can be made. A Request message comprises the piece index, offset, and requested data length, three integers, thus making a Request message 13 bytes (0x0D), as shown below are four Request requests:

[00 00 00 0D 06 00 00 00 00 00 00 00 00 00 00 10 00]
[00 00 00 0D 06 00 00 00 00 00 00 10 00 00 00 10 00]
[00 00 00 0D 06 00 00 00 00 00 00 20 00 00 00 10 00]
[00 00 00 0D 06 00 00 00 00 00 00 30 00 00 00 10 00]

Each of these data requests the piece at index 0, with offsets 0x00, 0x1000, 0x2000, and 0x3000 respectively, and the size of the requested data is 4096 bytes (0x1000).

After sending a Request message, the other party will return data via a Piece message, directly following the request length and tag with the offset and data, as follows:

Request and Piece Response

Take the first Piece response as an example:

[00 00 10 09 07 00 00 00 00 00 00 00 00 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71 72 73 74 75 76 77 78 79 ...

This data length is 4105 (0x1009) bytes, type is Piece (07), offset is 0 (0x0), with subsequent data matching the contents of the generated bin file.

Requested content can be requested again or setting different offsets, as follows:

Setting Different Offsets

By continuously requesting chunks, a piece can be completed. Once all pieces are complete, the original file can be reconstructed based on the Info section of the metadata. For a multi-file torrent, individual files can be retrieved by calculating the piece and offset where each file resides.

Others (Have, Cancel, Request)

After downloading a piece, notify all connections with the Have message, its content requiring the piece index in addition to the type tag.

To notify a peer to cancel upload, the structure of the Cancel message is the same as the Request message.

2. Becoming an Excellent BitTorrent User

An excellent BitTorrent user should adhere to the rules, promote sharing, avoid distributing malware, and refrain from abuse as basic etiquette.

The purpose of this article is to analyze and explain the data transmission content between nodes in BitTorrent through a simple example. Due to constraints of space, implementation difficulty, and balancing detail, monitoring local ports and providing uploads to other nodes were not included. In the actual community, users abruptly stopping upload right after download completion, known as “Hit and Run,” is viewed as a negative behavior within the BitTorrent community, and this behavior is explicitly opposed here.

The analysis of BitTorrent’s basic protocol (BEP 0003) concludes here. Based on this content, a substantial understanding of BitTorrent fundamentals allows for the building of many tools. In future articles (possibly), selected extended protocols will be analyzed. If future articles are available, the links will be placed here:

• Introduction to Bittorrent Protocol (Part 4) Distributed Hash