Protocol specification

From Bitmessage Wiki
Revision as of 09:23, 26 April 2014 by JonathanCoe (talk | contribs) (Minor change to description of v4 pubkeys)
Jump to navigation Jump to search

Common standards

Hashes

Most of the time SHA-512 hashes are used, however RIPEMD-160 is also used when creating an address.

A double-round of SHA-512 is used for the Proof Of Work. Example of double-SHA-512 encoding of string "hello":

hello
9b71d224bd62f3785d96d46ad3ea3d73319bfbc2890caadae2dff72519673ca72323c3d99ba5c11d7c7acc6e14b8c5da0c4663475c2e5c3adef46f73bcdec043(first round of sha-512)
0592a10584ffabf96539f3d780d776828c67da1ab5b169e9e8aed838aaecc9ed36d49ff1423c55f019e050c66c6324f53588be88894fef4dcffdb74b98e2b200(second round of sha-512)

For Bitmessage addresses (RIPEMD-160) this would give:

hello
9b71d224bd62f3785d96d46ad3ea3d73319bfbc2890caadae2dff72519673ca72323c3d99ba5c11d7c7acc6e14b8c5da0c4663475c2e5c3adef46f73bcdec043(first round is sha-512)
79a324faeebcbf9849f310545ed531556882487e (with ripemd-160)

Common structures

All integers are encoded in big endian. (This is different from Bitcoin).

Message structure

Field Size Description Data type Comments
4 magic uint32_t Magic value indicating message origin network, and used to seek to next message when stream state is unknown
12 command char[12] ASCII string identifying the packet content, NULL padded (non-NULL padding results in packet rejected)
4 length uint32_t Length of payload in number of bytes
4 checksum uint32_t First 4 bytes of sha512(payload)
? payload uchar[] The actual data, a message or an object

Known magic values:

Magic value Sent over wire as
0xE9BEB4D9 E9 BE B4 D9

Variable length integer

Integer can be encoded depending on the represented value to save space. Variable length integers always precede an array/vector of a type of data that may vary in length.

Value Storage length Format
< 0xfd 1 uint8_t
<= 0xffff 3 0xfd followed by the integer as uint16_t
<= 0xffffffff 5 0xfe followed by the integer as uint32_t
- 9 0xff followed by the integer as uint64_t

Variable length string

Variable length string can be stored using a variable length integer followed by the string itself.

Field Size Description Data type Comments
1+ length var_int Length of the string
? string char[] The string itself (can be empty)

Variable length list of integers

n integers can be stored using n+1 variable length integers where the first var_int equals n.

Field Size Description Data type Comments
1+ count var_int Number of var_ints below
1+ var_int The first value stored
1+ var_int The second value stored...
1+ var_int etc...

Network address

When a network address is needed somewhere, this structure is used. This protocol and structure supports IPv6, but note that the original client currently only supports IPv4 networking. Network addresses are not prefixed with a timestamp or stream in the version message.

Field Size Description Data type Comments
4 (or 8) time uint32 the Time. Protocol version 1 clients use 4 byte time while protocol version 2 clients use 8 byte time.
4 stream uint32 Stream number for this node
8 services uint64_t same service(s) listed in version
16 IPv6/4 char[16] IPv6 address. The original client only supports IPv4 and only reads the last 4 bytes to get the IPv4 address. However, the IPv4 address is written into the message as a 16 byte IPv4-mapped IPv6 address

(12 bytes 00 00 00 00 00 00 00 00 00 00 FF FF, followed by the 4 bytes of the IPv4 address).

2 port uint16_t port number

Inventory Vectors

Inventory vectors are used for notifying other nodes about objects they have or data which is being requested. Two rounds of SHA-512 are used, resulting in a 64 byte hash. Only the first 32 bytes are used; the later 32 bytes are ignored.

Inventory vectors consist of the following data format:

Field Size Description Data type Comments
32 hash char[32] Hash of the object

Encrypted payload

Bitmessage uses ECIES to encrypt its messages. For more information see Encryption

Field Size Description Data type Comments
16 IV uchar[] Initialization Vector used for AES-256-CBC
2 uint16_t Curve type Elliptic Curve type 0x02CA (714)
2 uint16_t X length Length of X component of public key R
X length uchar[] X X component of public key R
2 uint16_t Y length Length of Y component of public key R
Y length uchar[] Y Y component of public key R
? encrypted uchar[] Cipher text
32 MAC uchar[] HMACSHA256 Message Authentication Code

Unencrypted Message Data

Field Size Description Data type Comments
1+ msg_version var_int Message format version
1+ address_version var_int Sender's address version number. This is needed in order to calculate the sender's address to show in the UI, and also to allow for forwards compatible changes to the public-key data included below.
1+ stream var_int Sender's stream number
4 behavior bitfield uint32_t A bitfield of optional behaviors and features that can be expected from the node with this pubkey included in this msg message (the sender's pubkey).
64 public signing key uchar[] The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
64 public encryption key uchar[] The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
1+ nonce_trials_per_byte var_int Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 320 is the network minimum so any lower values will be automatically raised to 320. This field is new and is only included when the address_version >= 3.
1+ extra_bytes var_int Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 14000 is the network minimum so any lower values will be automatically raised to 14000. This field is new and is only included when the address_version >= 3.
20 destination ripe uchar[] The ripe hash of the public key of the receiver of the message
1+ encoding var_int Message Encoding type
1+ message_length var_int Message Length
message_length message uchar[] The message.
1+ ack_length var_int Length of the acknowledgement data
ack_length ack_data uchar[] The acknowledgement data to be transmitted. This takes the form of a Bitmessage protocol message, like another msg message. The POW therein must already be completed.
1+ sig_length var_int Length of the signature
sig_length signature uchar[] The ECDSA signature which covers everything from the msg_version to the ack_data.

Message Encodings

Value Name Description
0 IGNORE Any data with this number may be ignored. The sending node might simply be sharing its public key with you.
1 TRIVIAL UTF-8. No 'Subject' or 'Body' sections. Useful for simple strings of data, like URIs or magnet links.
2 SIMPLE UTF-8. Uses 'Subject' and 'Body' sections. No MIME is used.

messageToTransmit = 'Subject:' + subject + '\n' + 'Body:' + message

Further values for the message encodings can be decided upon by the community. Any MIME or MIME-like encoding format, should they be used, should make use of Bitmessage's 8-bit bytes.

Pubkey bitfield features

Bit Name Description
0 undefined The most significant bit at the beginning of the structure. Undefined
1 undefined The next most significant bit. Undefined
... ... ...
30 include_destination Receiving node expects that the RIPE hash encoded in their address preceedes the encrypted message data of msg messages bound for them.
31 does_ack If true, the receiving node does send acknowledgements (rather than dropping them).

Message types

version

When a node creates an outgoing connection, it will immediately advertise its version. The remote node will respond with its version. No futher communication is possible until both peers have exchanged their version.

Payload:

Field Size Description Data type Comments
4 version int32_t Identifies protocol version being used by the node
8 services uint64_t bitfield of features to be enabled for this connection
8 timestamp int64_t standard UNIX timestamp in seconds
26 addr_recv net_addr The network address of the node receiving this message (not including the time or stream number)
26 addr_from net_addr The network address of the node emitting this message (not including the time or stream number and the ip itself is ignored by the receiver)
8 nonce uint64_t Random nonce used to detect connections to self.
1+ user_agent var_str User Agent (0x00 if string is 0 bytes long)
1+ stream numbers var_int_list The stream numbers that the emitting node is interested in.

A "verack" packet shall be sent if the version packet was accepted. Once you have sent and received a verack messages with the remote node, send an addr message advertising up to 1000 peers of which you are aware, and one or more inv messages advertising all of the valid objects of which you are aware.

The following services are currently assigned:

Value Name Description
1 NODE_NETWORK This is a normal network node.

verack

The verack message is sent in reply to version. This message consists of only a message header with the command string "verack".

addr

Provide information on known nodes of the network. Non-advertised nodes should be forgotten after typically 3 hours

Payload:

Field Size Description Data type Comments
1+ count var_int Number of address entries (max: 1000)
34x? addr_list net_addr Address of other nodes on the network.

inv

Allows a node to advertise its knowledge of one or more objects. Payload (maximum payload length: 50000 items):

Field Size Description Data type Comments
? count var_int Number of inventory entries
32x? inventory inv_vect[] Inventory vectors

getdata

getdata is used in response to an inv message to retrieve the content of a specific object after filtering known elements.

Payload (maximum payload length: 50000 entries):

Field Size Description Data type Comments
? count var_int Number of inventory entries
32x? inventory inv_vect[] Inventory vectors

Object types

Objects are a subset of network messages. They are shared throughout a stream. A client should advertise objects that are not older than 2.5 days. To be a valid object, the Proof Of Work has to be done.

getpubkey

When a node has the hash of a public key (from an address) but not the public key itself, it must send out a request for the public key.

Field Size Description Data type Comments
8 POW nonce uint64_t Random nonce used for the Proof Of Work
4 (or 8) time uint32_t The time that this message was generated and broadcast. We are transitioning to 8 byte time.
1+ address version var_int The address' version
1+ stream number var_int The address' stream number
20 ripe uchar[] The ripemd hash of the public key. This field is only included when the address version is <= 3.
32 tag uchar[] The tag derived from the address version, stream number, and ripe. This field is only included when the address version is >= 4.

pubkey

A version 2 pubkey. This is still in use and supported by current clients but new v2 addresses are not generated by clients.

Field Size Description Data type Comments
8 POW nonce uint64_t Random nonce used for the Proof Of Work
4 (or 8) time uint32_t The time that this pubkey was generated and broadcast. We are transitioning to 8 byte time.
1+ address version var_int The address' version
1+ stream number var_int The address' stream number
4 behavior bitfield uint32_t A bitfield of optional behaviors and features that can be expected from the node receiving the message.
64 public signing key uchar[] The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
64 public encryption key uchar[] The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )

A version 3 pubkey

Field Size Description Data type Comments
8 POW nonce uint64_t Random nonce used for the Proof Of Work
4 (or 8) time uint32_t The time that this pubkey was generated and broadcast. We are transitioning to 8 byte time.
1+ address version var_int The address' version which is set to 3.
1+ stream number var_int The address' stream number
4 behavior bitfield uint32_t A bitfield of optional behaviors and features that can be expected from the node receiving the message.
64 public signing key uchar[] The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
64 public encryption key uchar[] The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
1+ nonce_trials_per_byte var_int Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 320 is the network minimum so any lower values will be automatically raised to 320.
1+ extra_bytes var_int Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 14000 is the network minimum so any lower values will be automatically raised to 14000.
1+ sig_length var_int Length of the signature
sig_length signature uchar[] The ECDSA signature which covers everything from the time to the extra_bytes.

A version 4 pubkey

Field Size Description Data type Comments
8 POW nonce uint64_t Random nonce used for the Proof Of Work
4 (or 8) time uint32_t The time that this pubkey was generated and broadcast. We are transitioning to 8 byte time.
1+ address version var_int The address' version
1+ stream number var_int The address' stream number
32 tag uchar[] The tag, made up of bytes 32-64 of the double hash of the address data (see example python code below)
? encrypted uchar[] Encrypted pubkey data.

When version 4 pubkeys are created, most of the data in the pubkey is encrypted. This is done in such a way that only someone who has the Bitmessage address which corresponds to a pubkey can decrypt and use that pubkey. This prevents people from gathering pubkeys sent around the network and using the data from them to create messages to be used in spam or in flooding attacks.

In order to encrypt the pubkey data, a double SHA-512 hash is calculated from the address version number, stream number, and ripe hash of the Bitmessage address that the pubkey corresponds to. The first 32 bytes of this hash are used as the key to encrypt the pubkey data. The remaining 32 bytes of this hash are added to the unencrypted part of the pubkey and used as a tag, as above. This allows nodes to determine which pubkey to decrypt when they wish to send a message.

In PyBitmessage, the double hash of the address data is calculated using the python code below:

doubleHashOfAddressData = hashlib.sha512(hashlib.sha512(encodeVarint(addressVersionNumber) + encodeVarint(streamNumber) + hash).digest()).digest()

Encrypted data in version 4 pubkeys:

Field Size Description Data type Comments
4 behavior bitfield uint32_t A bitfield of optional behaviors and features that can be expected from the node receiving the message.
64 public signing key uchar[] The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
64 public encryption key uchar[] The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
1+ nonce_trials_per_byte var_int Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 320 is the network minimum so any lower values will be automatically raised to 320.
1+ extra_bytes var_int Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 14000 is the network minimum so any lower values will be automatically raised to 14000.
1+ sig_length var_int Length of the signature
sig_length signature uchar[] The ECDSA signature which covers everything from the time to the extra_bytes.

msg

Used for person-to-person messages.

Field Size Description Data type Comments
8 POW nonce uint64_t Random nonce used for the Proof Of Work
4 (or 8) time uint32_t The time that this message was generated and broadcast. We are transitioning to 8 byte time.
1+ streamNumber var_int The stream number of the destination address.
? encrypted uchar[] Encrypted data. See Encrypted payload. See also Unencrypted Message Data Format

broadcast

Users who are subscribed to the sending address will see the message appear in their inbox.

Pubkey objects and v3 broadcast objects are encrypted the same way: The data encoded in the sender's Bitmessage address is hashed twice. The first 32 bytes of the resulting hash constitutes the "private" encryption key and the last 32 bytes constitute a tag so that anyone listening can easily decide if this particular message is interesting. The sender calculates the public key from the private key and then encrypts the object with this public key. Thus anyone who knows the Bitmessage address of the sender of a broadcast or pubkey object can decrypt it.

Field Size Description Data type Comments
8 POW nonce uint64_t The Proof Of Work nonce
8 time uint64_t The time that the message was broadcast.
1+ broadcast version var_int The version number of this broadcast protocol message which is equal to 2 or 3.
1+ stream number var_int The sender's stream number
32 tag uchar[] The tag. This field is new and only included when the broadcast version is >= 3.
? encrypted uchar[] Encrypted broadcast data. The keys are derived as described in the paragraph above. See Encrypted payload for details about the encryption algorithm itself.

Unencrypted data format:

Field Size Description Data type Comments
1+ broadcast version var_int The version number of this broadcast protocol message which is equal to 2 or 3. This is included here so that it can be signed.
1+ address version var_int The sender's address version
1+ stream number var_int The sender's stream number
4 behavior bitfield uint32_t A bitfield of optional behaviors and features that can be expected from the owner of this pubkey.
64 public signing key uchar[] The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
64 public encryption key uchar[] The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
1+ nonce_trials_per_byte var_int Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 320 is the network minimum so any lower values will be automatically raised to 320. This field is new and is only included when the address_version >= 3.
1+ extra_bytes var_int Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 14000 is the network minimum so any lower values will be automatically raised to 14000. This field is new and is only included when the address_version >= 3.
1+ encoding var_int The encoding type of the message
1+ messageLength var_int The message length in bytes
messageLength message uchar[] The message
1+ sig_length var_int Length of the signature
sig_length signature uchar[] The signature which covers everything from the broadcast version down through the message.