Difference between revisions of "Protocol specification v3"

From Bitmessage Wiki
Jump to navigation Jump to search
(version)
 
(48 intermediate revisions by 3 users not shown)
Line 4: Line 4:
  
 
== Introduction ==
 
== Introduction ==
This is a DRAFT for the protocol version 3. It describes the changes in protocol version 3 versus version 2. Things which are unchanged from version 2 are not described in detail. So you should use [https://bitmessage.org/wiki/Protocol_specification this] as a reference for all formates which are not mentioned in this description.
+
This is a DRAFT for the protocol version 3. It describes the changes in protocol version 3 versus version 2. Things which are unchanged from version 2 are not described in detail. So you should use the [[Protocol specification|current protocol specification]], which includes the version 3 changes, as a reference for all formats which are not mentioned in this description.
 +
 
 +
== New features in version 3 ==
 +
 
 +
Here are the new features of the version 3 of Bitmessage protocol in keywords:
 +
 
 +
* object type is now coded inside the message
 +
* objects may have a variable time to live
 +
* The POW is more difficult but can be easier if you lower the time to live
 +
* The protocol is tolerant for further message extension
 +
* The protocol is tolerant for further object extension
 +
* A lower maximum object size
 +
 
 +
 
 +
== Common structures ==
 +
 
 +
=== Variable length integer ===
 +
 
 +
The shortest possible encoding MUST be used. In version 2 the decimal value 10 could be encoded as 0x0A, 0xFD000A, 0xFE0000000A, or 0xFF000000000000000A. In version 3 only the shortest representation, 0x0A, is allowed. The 9-byte form is no longer useful and SHOULD NOT be used.
 +
 
 +
Here is an example address which uses malformed varints: BM-CVA3RC7Mvy7JDNSpQChktwrSe4KNMaEdDdcymfUo. This address is invalid.
 +
 
  
 
== Message types ==
 
== Message types ==
  
Most message types are unchanged from version 2 to version 3. Only the four "objecttype" messages are not valid any more. They are summerized into one single message.
+
Most message types are unchanged from version 2 to version 3. Only the four "objecttype" messages are not valid any more. They are summarized into one single message.
  
 
=== version ===
 
=== version ===
  
The version message is identically to protocol version 2. you can lookup details [https://bitmessage.org/wiki/Protocol_specification#version here]
+
The version message is identical to [[Protocol specification#version|version 2 version message]].
 +
 
 +
=== verack ===
  
Payload:
+
The verack message is identical to [[Protocol specification#verack|version 2 verack message]].
 +
 
 +
=== addr ===
 +
 
 +
The addr message is identical to [[Protocol specification#addr|version 2 addr message]].
 +
 
 +
=== inv ===
 +
 
 +
The inv message is identical to [[Protocol specification#inv|version 2 inv message]].
 +
 
 +
=== getdata ===
 +
 
 +
The getdata message is identical to [[Protocol specification#getdata|version 2 getdata message]].
 +
 
 +
=== error ===
 +
 
 +
Version 3 of the protocol defined a special error (or debug) message. This message may be silently ignored (and therefor handled like any other "unknown" message).
 +
The message is intended to inform the other node about protocol errors and can be used for debugging and improving code.
  
 
{|class="wikitable"
 
{|class="wikitable"
 
! Field Size !! Description !! Data type !! Comments
 
! Field Size !! Description !! Data type !! Comments
 
|-
 
|-
| 4 || version || int32_t || Identifies protocol version being used by the node
+
| 1+||fatal||var_int||
 +
This qualifies the error. If set to 0, than its just a "warning". You can expect, everything still worked fine. If set to 1, than it's an error, so you may expect, something was going wrong (e.g. an object got lost). If set to 2, it's a fatal error. The node will drop the line for that error and maybe ban you for some time.
 
|-
 
|-
| 8 || services || uint64_t || bitfield of features to be enabled for this connection
+
 
 
|-
 
|-
| 8 || timestamp || int64_t || standard UNIX timestamp in seconds
+
| 1+||ban time||var_int||
 +
If the error is fatal, you can specify the ban time in seconds, here. You inform the other node, that you will not accept further connections for this number of seconds. For non fatal errors this field has no meaning and should be zero.
 
|-
 
|-
| 26 || addr_recv || net_addr || The network address of the node receiving this message (not including the time or stream number)
+
 
 
|-
 
|-
| 26 || addr_from || net_addr || The network address of the node emitting this message (not including the time or stream number and the ip itself is ignored by the receiver)
+
| 1+||inventory vector||var_str||
 +
If the error is related to an object, this Variable length string contains the inventory vector of that object. If the error is not related to an object, this string is empty.
 
|-
 
|-
| 8 || nonce || uint64_t || Random nonce used to detect connections to self.
+
 
 
|-
 
|-
| 1+ || user_agent || [[#Variable length string|var_str]] || [[User Agent]] (0x00 if string is 0 bytes long)
+
| 1+||error text||var_str||
 +
A human readable string in English, which describes the error.
 
|-
 
|-
| 1+ || stream numbers || [[#Variable length list of integers|var_int_list]] || The stream numbers that the emitting node is interested in.
+
 
 
|}
 
|}
  
A "verack" packet shall be sent if the version packet was accepted. Once you have sent and received a verack messages with the remote node, send an addr message advertising up to 1000 peers of which you are aware, and one or more inv messages advertising all of the valid objects of which you are aware.
+
=== Unsupported messages ===
 +
If a node receives an unknown message it <u>must</u> silently ignore it. This is for further extensions of the protocol with other messages. Nodes that don't understand such a new message type shall be able to work correct with the message types they understand.
  
The following services are currently assigned:
+
Maybe some version 2 nodes did already implement it that way, but in version 3 it is '''part of the protocol specification''', that a node <u>must</u> silently ignore unsupported messages.
  
{|class="wikitable"
+
== Object type ==
! Value !! Name !! Description
 
|-
 
| 1 || NODE_NETWORK || This is a normal network node.
 
|}
 
  
=== verack ===
+
The four former object types "getpubkey", "pubkey", "msg" and "broadcast" are summarized into a single Message type "object". The four Messages as they did exist in version 2 protocol are not valid any more.
  
The ''verack'' message is sent in reply to ''version''This message consists of only a [[#Message structure|message header]] with the command string "verack". The TCP timeout starts out at 20 seconds; after verack messages are exchanged, the timeout is raised to 10 minutes.
+
Objects are shared throughout a streamA client should advertise objects until their end of life time is reached. To be a valid object, the [[Proof of work|Proof Of Work]] has to be done.
  
=== addr ===
 
 
Provide information on known nodes of the network. Non-advertised nodes should be forgotten after typically 3 hours
 
 
Payload:
 
 
{|class="wikitable"
 
{|class="wikitable"
 
! Field Size !! Description !! Data type !! Comments
 
! Field Size !! Description !! Data type !! Comments
 
|-
 
|-
| 1+ || count || [[Protocol_specification#Variable_length_integer|var_int]] || Number of address entries (max: 1000)
+
| 8||POW nonce||uint64_t||
 +
Random nonce used for the [[Proof of work|Proof Of Work]]
 +
|-
 +
| 8||time||uint64_t||
 +
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be broadcast until its end of life time has been reached. The node shall store the inventory vector of that object for at least another hour to avoid reloading it from another node with a small time delay.
 +
The maximum value for the "time to life" of an object is 28 days. so the "end of life time" is 28 days in the future at maximum.
 +
|-
 +
| 4||object type||uint32_t||
 +
This field specifies the content of the object.
 +
Valid values are (at the moment) 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". all other values are reserved. Nodes shall transport objects with unknown types, too. This will make further protocol changes more easy.
 +
|-
 +
|-
 +
| ?||payload||uchar[]||
 +
This field varies depending on the object type. For a detailed description of their content refer to [[Protocol specification#Object_types|version 2 object types]]
 
|-
 
|-
| 34x? || addr_list || [[Protocol_specification#Network_address|net_addr]] || Address of other nodes on the network.
 
 
|}
 
|}
  
=== inv ===
 
  
Allows a node to advertise its knowledge of one or more objects.  
+
== Proof of Work ==
Payload (maximum payload length: 50000 items):
+
 
 +
Generally the POW is done exactly like in [[Proof of work|version 2]]
 +
 
 +
The "target" (the difficulty of the POW) is defined a little bit lower (more difficult) in version 3. This is, because practice did show, it is to easy to flood the network with data.
 +
In addition to that it is possible in version 3 to lower the time to live of a message (for example when doing a live chat) and getting an easier POW for that.
 +
 
 +
 
 +
[[File:POW forumla 2.png|none|caption]]
  
{|class="wikitable"
 
! Field Size !! Description !! Data type !! Comments
 
|-
 
| ? || count || [[Protocol_specification#Variable_length_integer|var_int]] || Number of inventory entries
 
|-
 
| 32x? || inventory || [[Protocol specification#Inventory Vectors|inv_vect]][] || Inventory vectors
 
|}
 
  
=== getdata ===
+
payloadLengthExtraBytes = 1000
  
getdata is used in response to an ''inv'' message to retrieve the content of a specific object after filtering known elements.
+
nonceTrialsPerByte = 1000
  
Payload (maximum payload length: 50000 entries):
+
== Maximum object size ==
  
{|class="wikitable"
+
In version 2 the maximum object size was defined to be 170 MBytes. This object size is totally unrealistic for a normal use (POW), but is perfectly for a network attack. It will be to big to handle for clients with low bandwidth.
! Field Size !! Description !! Data type !! Comments
+
Currently an average bitmessage "msg" object has the size of 2 kBytes.
|-
+
So version 3 limits the objects to a maximum size of 256 KiB(the payload of the object, starting from the POW nonce can have 262144 bytes at maximum).
| ? || count || [[Protocol_specification#Variable_length_integer|var_int]] || Number of inventory entries
 
|-
 
| 32x? || inventory || [[Protocol specification#Inventory Vectors|inv_vect]][] || Inventory vectors
 
|}
 

Latest revision as of 20:14, 17 October 2014

Introduction

This is a DRAFT for the protocol version 3. It describes the changes in protocol version 3 versus version 2. Things which are unchanged from version 2 are not described in detail. So you should use the current protocol specification, which includes the version 3 changes, as a reference for all formats which are not mentioned in this description.

New features in version 3

Here are the new features of the version 3 of Bitmessage protocol in keywords:

  • object type is now coded inside the message
  • objects may have a variable time to live
  • The POW is more difficult but can be easier if you lower the time to live
  • The protocol is tolerant for further message extension
  • The protocol is tolerant for further object extension
  • A lower maximum object size


Common structures

Variable length integer

The shortest possible encoding MUST be used. In version 2 the decimal value 10 could be encoded as 0x0A, 0xFD000A, 0xFE0000000A, or 0xFF000000000000000A. In version 3 only the shortest representation, 0x0A, is allowed. The 9-byte form is no longer useful and SHOULD NOT be used.

Here is an example address which uses malformed varints: BM-CVA3RC7Mvy7JDNSpQChktwrSe4KNMaEdDdcymfUo. This address is invalid.


Message types

Most message types are unchanged from version 2 to version 3. Only the four "objecttype" messages are not valid any more. They are summarized into one single message.

version

The version message is identical to version 2 version message.

verack

The verack message is identical to version 2 verack message.

addr

The addr message is identical to version 2 addr message.

inv

The inv message is identical to version 2 inv message.

getdata

The getdata message is identical to version 2 getdata message.

error

Version 3 of the protocol defined a special error (or debug) message. This message may be silently ignored (and therefor handled like any other "unknown" message). The message is intended to inform the other node about protocol errors and can be used for debugging and improving code.

Field Size Description Data type Comments
1+ fatal var_int

This qualifies the error. If set to 0, than its just a "warning". You can expect, everything still worked fine. If set to 1, than it's an error, so you may expect, something was going wrong (e.g. an object got lost). If set to 2, it's a fatal error. The node will drop the line for that error and maybe ban you for some time.

1+ ban time var_int

If the error is fatal, you can specify the ban time in seconds, here. You inform the other node, that you will not accept further connections for this number of seconds. For non fatal errors this field has no meaning and should be zero.

1+ inventory vector var_str

If the error is related to an object, this Variable length string contains the inventory vector of that object. If the error is not related to an object, this string is empty.

1+ error text var_str

A human readable string in English, which describes the error.

Unsupported messages

If a node receives an unknown message it must silently ignore it. This is for further extensions of the protocol with other messages. Nodes that don't understand such a new message type shall be able to work correct with the message types they understand.

Maybe some version 2 nodes did already implement it that way, but in version 3 it is part of the protocol specification, that a node must silently ignore unsupported messages.

Object type

The four former object types "getpubkey", "pubkey", "msg" and "broadcast" are summarized into a single Message type "object". The four Messages as they did exist in version 2 protocol are not valid any more.

Objects are shared throughout a stream. A client should advertise objects until their end of life time is reached. To be a valid object, the Proof Of Work has to be done.

Field Size Description Data type Comments
8 POW nonce uint64_t

Random nonce used for the Proof Of Work

8 time uint64_t

The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be broadcast until its end of life time has been reached. The node shall store the inventory vector of that object for at least another hour to avoid reloading it from another node with a small time delay. The maximum value for the "time to life" of an object is 28 days. so the "end of life time" is 28 days in the future at maximum.

4 object type uint32_t

This field specifies the content of the object. Valid values are (at the moment) 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". all other values are reserved. Nodes shall transport objects with unknown types, too. This will make further protocol changes more easy.

? payload uchar[]

This field varies depending on the object type. For a detailed description of their content refer to version 2 object types


Proof of Work

Generally the POW is done exactly like in version 2

The "target" (the difficulty of the POW) is defined a little bit lower (more difficult) in version 3. This is, because practice did show, it is to easy to flood the network with data. In addition to that it is possible in version 3 to lower the time to live of a message (for example when doing a live chat) and getting an easier POW for that.


caption


payloadLengthExtraBytes = 1000

nonceTrialsPerByte = 1000

Maximum object size

In version 2 the maximum object size was defined to be 170 MBytes. This object size is totally unrealistic for a normal use (POW), but is perfectly for a network attack. It will be to big to handle for clients with low bandwidth. Currently an average bitmessage "msg" object has the size of 2 kBytes. So version 3 limits the objects to a maximum size of 256 KiB(the payload of the object, starting from the POW nonce can have 262144 bytes at maximum).