Bitmessage Wiki - User contributions [en]

Developer Reference

2015-02-09T12:09:45Z

JonathanCoe: Added mailchuck.com

This page is intended to provide information and other resources that are useful for Bitmessage developers.

==Protocol Information==

Protocol specification: https://bitmessage.org/wiki/Protocol_specification

Encryption scheme: https://bitmessage.org/wiki/Encryption

Proof of Work scheme: https://bitmessage.org/wiki/Proof_of_work

Bitmessage White Paper: https://bitmessage.org/bitmessage.pdf

Bitmessage Technical Paper (Note: not updated for protocol version 3): https://bitmessage.org/Bitmessage%20Technical%20Paper.pdf

== Implementations ==

===Full Node Implementations===

PyBitmessage (Reference Client) (Python): https://github.com/bitmessage/pybitmessage

bitmessaged (C++): https://github.com/Thomas-Astade/bitmessaged

===Lite Client Implementations===

Bitseal (Java): https://github.com/jonathancoe/bitseal

Notbit (C): https://github.com/bpeel/notbit

Bmr (Javascript): https://github.com/chovy/bmr

bitmessage-web (Javascript): https://github.com/jclement/bitmessage-web

Bitpost (Objective-C): https://github.com/Voluntarynet/Bitpost

===Web Clients===

Blinked (Javascript): https://blinked.ca

Bitmsg.me (Javascript): https://bitmsg.me

===Gateway Services===

https://bitmessage.ch

http://bitmessage.mobi

https://mailchuck.com

===Other Implementations===

Please note that some of these other implementations may be incomplete or not up-to-date with the current Bitmessage protocol.

libbitmessage (C++): https://github.com/corebob/libbitmessage

bitmessage-go (Go): https://github.com/corebob/bitmessage-go

cppbitmessage (C++): https://github.com/bashrc/cppbitmessage

JBitmessage (Java): https://github.com/ISibboI/JBitmessage

SharpBitmessage (C#): https://github.com/sharpbitmessage/SharpBitmessage

Bitmessage-js (Javascript): https://github.com/indigots/Bitmessage-js

bitmessage-ruby (Ruby): https://github.com/staii/bitmessage-ruby

bitchan (Javascript): https://github.com/bitchan

==Scripts and Utilities==

===PyBitmessage Utilities===

Bitmessage PHP class - Bitmessage PHP Class to control PyBitmessage daemon using xmlrpc - https://github.com/Conver/class.bitmessage.php

bmwrapper - Email wrapper for PyBitmessage: https://github.com/Arceliar/bmwrapper

BitCrypt - Encrypts and decrypts PyBitmessage .dat files: https://github.com/AyrA/BitCrypt

BitUpdate - Automatically update PyBitmessage: https://github.com/AyrA/BitUpdate

NIIP-BitMessageLib - A .NET implementation of the API exposed by PyBitmessage: https://bitbucket.org/niip/niip-bitmessagelib

===Others===

bitmessage-powfaster - Bitmessage Proof Of Work optimizations including OpenCL and C based PoW code: https://github.com/grant-olson/bitmessage-powfaster

BitMailServer - Bitmesssage to Email Gateway: https://github.com/AyrA/BitMailServer

BitDNS - Bitmessage DNS and Namecoin integration: https://github.com/AyrA/BitDNS

BitCenter - Powerful Bitmessage message processing: https://github.com/AyrA/BitCenter

BitHTTP - HTTP proxy over Bitmessage: https://github.com/AyrA/BitHTTP

BinSend - Send and decode binary attachments via Bitmessage: https://github.com/AyrA/BinSend

BitMessageForum - Browse your bitmessages via a forum-like UI: https://github.com/grant-olson/BitMessageForum

Developer Reference

2015-02-09T11:54:04Z

JonathanCoe: Improved spacing

This page is intended to provide information and other resources that are useful for Bitmessage developers.

==Protocol Information==

Protocol specification: https://bitmessage.org/wiki/Protocol_specification

Encryption scheme: https://bitmessage.org/wiki/Encryption

Proof of Work scheme: https://bitmessage.org/wiki/Proof_of_work

Bitmessage White Paper: https://bitmessage.org/bitmessage.pdf

Bitmessage Technical Paper (Note: not updated for protocol version 3): https://bitmessage.org/Bitmessage%20Technical%20Paper.pdf

== Implementations ==

===Full Node Implementations===

PyBitmessage (Reference Client) (Python): https://github.com/bitmessage/pybitmessage

bitmessaged (C++): https://github.com/Thomas-Astade/bitmessaged

===Lite Client Implementations===

Bitseal (Java): https://github.com/jonathancoe/bitseal

Notbit (C): https://github.com/bpeel/notbit

Bmr (Javascript): https://github.com/chovy/bmr

bitmessage-web (Javascript): https://github.com/jclement/bitmessage-web

Bitpost (Objective-C): https://github.com/Voluntarynet/Bitpost

===Web Clients===

Blinked (Javascript): https://blinked.ca

Bitmsg.me (Javascript): https://bitmsg.me

===Gateway Services===

https://bitmessage.ch

http://bitmessage.mobi

===Other Implementations===

Please note that some of these other implementations may be incomplete or not up-to-date with the current Bitmessage protocol.

libbitmessage (C++): https://github.com/corebob/libbitmessage

bitmessage-go (Go): https://github.com/corebob/bitmessage-go

cppbitmessage (C++): https://github.com/bashrc/cppbitmessage

JBitmessage (Java): https://github.com/ISibboI/JBitmessage

SharpBitmessage (C#): https://github.com/sharpbitmessage/SharpBitmessage

Bitmessage-js (Javascript): https://github.com/indigots/Bitmessage-js

bitmessage-ruby (Ruby): https://github.com/staii/bitmessage-ruby

bitchan (Javascript): https://github.com/bitchan

==Scripts and Utilities==

===PyBitmessage Utilities===

Bitmessage PHP class - Bitmessage PHP Class to control PyBitmessage daemon using xmlrpc - https://github.com/Conver/class.bitmessage.php

bmwrapper - Email wrapper for PyBitmessage: https://github.com/Arceliar/bmwrapper

BitCrypt - Encrypts and decrypts PyBitmessage .dat files: https://github.com/AyrA/BitCrypt

BitUpdate - Automatically update PyBitmessage: https://github.com/AyrA/BitUpdate

NIIP-BitMessageLib - A .NET implementation of the API exposed by PyBitmessage: https://bitbucket.org/niip/niip-bitmessagelib

===Others===

bitmessage-powfaster - Bitmessage Proof Of Work optimizations including OpenCL and C based PoW code: https://github.com/grant-olson/bitmessage-powfaster

BitMailServer - Bitmesssage to Email Gateway: https://github.com/AyrA/BitMailServer

BitDNS - Bitmessage DNS and Namecoin integration: https://github.com/AyrA/BitDNS

BitCenter - Powerful Bitmessage message processing: https://github.com/AyrA/BitCenter

BitHTTP - HTTP proxy over Bitmessage: https://github.com/AyrA/BitHTTP

BinSend - Send and decode binary attachments via Bitmessage: https://github.com/AyrA/BinSend

BitMessageForum - Browse your bitmessages via a forum-like UI: https://github.com/grant-olson/BitMessageForum

Developer Reference

2015-02-09T11:53:43Z

JonathanCoe: Added bitchan and NIIP-BitMessageLib

This page is intended to provide information and other resources that are useful for Bitmessage developers.

==Protocol Information==

Protocol specification: https://bitmessage.org/wiki/Protocol_specification

Encryption scheme: https://bitmessage.org/wiki/Encryption

Proof of Work scheme: https://bitmessage.org/wiki/Proof_of_work

Bitmessage White Paper: https://bitmessage.org/bitmessage.pdf

Bitmessage Technical Paper (Note: not updated for protocol version 3): https://bitmessage.org/Bitmessage%20Technical%20Paper.pdf

== Implementations ==

===Full Node Implementations===

PyBitmessage (Reference Client) (Python): https://github.com/bitmessage/pybitmessage

bitmessaged (C++): https://github.com/Thomas-Astade/bitmessaged

===Lite Client Implementations===

Bitseal (Java): https://github.com/jonathancoe/bitseal

Notbit (C): https://github.com/bpeel/notbit

Bmr (Javascript): https://github.com/chovy/bmr

bitmessage-web (Javascript): https://github.com/jclement/bitmessage-web

Bitpost (Objective-C): https://github.com/Voluntarynet/Bitpost

===Web Clients===

Blinked (Javascript): https://blinked.ca

Bitmsg.me (Javascript): https://bitmsg.me

===Gateway Services===

https://bitmessage.ch

http://bitmessage.mobi

===Other Implementations===

Please note that some of these other implementations may be incomplete or not up-to-date with the current Bitmessage protocol.

libbitmessage (C++): https://github.com/corebob/libbitmessage

bitmessage-go (Go): https://github.com/corebob/bitmessage-go

cppbitmessage (C++): https://github.com/bashrc/cppbitmessage

JBitmessage (Java): https://github.com/ISibboI/JBitmessage

SharpBitmessage (C#): https://github.com/sharpbitmessage/SharpBitmessage

Bitmessage-js (Javascript): https://github.com/indigots/Bitmessage-js

bitmessage-ruby (Ruby): https://github.com/staii/bitmessage-ruby

bitchan (Javascript): https://github.com/bitchan

==Scripts and Utilities==

===PyBitmessage Utilities===

Bitmessage PHP class - Bitmessage PHP Class to control PyBitmessage daemon using xmlrpc - https://github.com/Conver/class.bitmessage.php

bmwrapper - Email wrapper for PyBitmessage: https://github.com/Arceliar/bmwrapper

BitCrypt - Encrypts and decrypts PyBitmessage .dat files: https://github.com/AyrA/BitCrypt

BitUpdate - Automatically update PyBitmessage: https://github.com/AyrA/BitUpdate

NIIP-BitMessageLib - A .NET implementation of the API exposed by PyBitmessage: https://bitbucket.org/niip/niip-bitmessagelib

===Others===

bitmessage-powfaster - Bitmessage Proof Of Work optimizations including OpenCL and C based PoW code: https://github.com/grant-olson/bitmessage-powfaster

BitMailServer - Bitmesssage to Email Gateway: https://github.com/AyrA/BitMailServer

BitDNS - Bitmessage DNS and Namecoin integration: https://github.com/AyrA/BitDNS

BitCenter - Powerful Bitmessage message processing: https://github.com/AyrA/BitCenter

BitHTTP - HTTP proxy over Bitmessage: https://github.com/AyrA/BitHTTP

BinSend - Send and decode binary attachments via Bitmessage: https://github.com/AyrA/BinSend

BitMessageForum - Browse your bitmessages via a forum-like UI: https://github.com/grant-olson/BitMessageForum

Scalability through Prefix Filtering

2015-02-03T13:30:07Z

JonathanCoe: Made spacing more consistent

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network also has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* As the network grows or shrinks, nodes also move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* Nodes maintain long-lived connections to nodes in their own stream and streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines which streams the object will propagate in.
* Nodes process objects in their own stream and all lower streams of the same branch.
* Objects propagate in the stream that matches their prefix nonce and all higher streams of the same branch.

==Background==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals. In general the problems shared by most stream systems are simplified. For example, instead the network having to create or abandon streams as the total network traffic level changes, nodes simply adjust their position in a pre-determined hierarchy of streams, moving either 'up' or 'down' the hierarchy.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion possible 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing up to 100,000 objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects per day then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

===Proposed Stream Structure===

The diagram below shows the structure of streams under the proposed system. The small, light blue circles represent groups of nodes. The large coloured circles represent streams. Note that this diagram only shows 4 complete levels (5 levels of nodes and 4 levels of streams). The proposed stream hierarchy would have 64 such levels.

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

===Node Connections===

The diagram below illustrates the connections between nodes and clients under the proposed system. Both nodes and clients are labelled with a stream number, which correspond to the stream numbers in the stream structure diagram above. Note that this diagram only shows nodes and clients in the first 4 levels of streams. In practice there would be 64 levels of streams.

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Client Procedures ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a higher stream of the same branch.
* The pubkey propagates through its address's stream and all higher streams of the same branch.

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client, or a node acting on behalf of the client, connects to one or more nodes with a stream number that matches any number of bits of the address's prefix value.
* The client then creates a getpubkey object with a prefix nonce that exactly matches the address's prefix value and sends it to those nodes.
* The client, or a node acting on its behalf, then repeatedly queries nodes in that stream for the pubkey in question until it is retrieved.

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's prefix.
* Sets the remaining bits of the msg prefix nonce randomly.
* Sends the msg to one or more nodes with a stream number matching the destination address’s identifying prefix bits or a higher steam.

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* For each address owned by the client, the client connects to one or more nodes in the address's stream or a higher stream of the same branch.
* The client then requests any objects with a prefix nonce where the first n bits match the destination address's stream number, where n is the prefix length value of the address.
* The client then processes these objects.

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs).
When a client wants to subscribe to broadcasts from an address, it does the following:
* Extracts the address's prefix value from the address.
* Retrieves the address's pubkey (see above).
* Extracts the prefix length value from the address's pubkey and uses it to calculate the address's current stream number.
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address that the client is subscribed to.

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 (shown as "stream _" in the stream hierarchy diagram) will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 4 would mean that the node will deal with 6.25% of the total network object traffic.
* A prefix length value of 8 would mean that the node will deal with 0.39% of the total network object traffic.
* A prefix length value of 14 would mean that the node will deal with 0.006% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, each node would keep a large list of the addresses of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. The IP addresses and stream numbers of new nodes would be spread widely through the network, regardless of the stream system.

Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream.

Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How can this be done in a way that does not undermine the security of the network and its users?

The basic idea is that nodes can measure the level of object traffic in their stream. All objects must have POW done for them, so the object traffic is valuable because it is costly to spoof. Nodes would move to lower streams when the level of object traffic in their stream is too great for them to handle and move to higher streams when the level of object traffic is below 50% of what they are capable of processing.

In order to prevent attackers 'pushing' nodes into lower streams by creating short bursts of very high volume object traffic, nodes could implement a time delay mechanism that ensures that they will only move up or down based on the object traffic levels of a relatively long period of time (e.g. 1 week).

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How can this be done in a way that does not undermine the security of the network and its users?

Similarly to nodes, addresses must move to higher or lower streams in order to preserve the balance between anonymity and efficiency that the address's owner desires. Again, it would make sense to use the level of object traffic to decide when to do this. As clients must download all objects in their address's stream(s), they should also be able to use the levels of object traffic to determine when each address needs to move up or down in the stream hierarchy.

Clients that control addresses could also implement a time delay mechanism, similar to the one described above for nodes.

Scalability through Prefix Filtering

2015-02-03T13:29:01Z

JonathanCoe: Minor correction

Scalability through Prefix Filtering

2015-02-03T13:19:36Z

JonathanCoe: Made some corrections

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network also has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* As the network grows or shrinks, nodes also move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* Nodes maintain long-lived connections to nodes in their own stream and streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines which streams the object will propagate in.
* Nodes process objects in their own stream and all lower streams of the same branch.
* Objects propagate in the stream that matches their prefix nonce and all higher streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals. In general the problems shared by most stream systems are simplified. For example, instead the network having to create or abandon streams as the total network traffic level changes, nodes simply adjust their position in a pre-determined hierarchy of streams, moving either 'up' or 'down' the hierarchy.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion possible 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing up to 100,000 objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects per day then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

===Proposed Stream Structure===

The diagram below shows the structure of streams under the proposed system. The small, light blue circles represent groups of nodes. The large coloured circles represent streams. Note that this diagram only shows 4 complete levels (5 levels of nodes and 4 levels of streams). The proposed stream hierarchy would have 64 such levels.

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

===Node Connections===

The diagram below illustrates the connections between nodes and clients under the proposed system. Both nodes and clients are labelled with a stream number, which correspond to the stream numbers in the stream structure diagram above. Note that this diagram only shows nodes and clients in the first 4 levels of streams. In practice there would be 64 levels of streams.

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Client Procedures ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a higher stream of the same branch.
* The pubkey propagates through its address's stream and all higher streams of the same branch.

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client, or a node acting on behalf of the client, connects to one or more nodes with a stream number that matches any number of bits of the address's prefix value.
* The client then creates a getpubkey object with a prefix nonce that exactly matches the address's prefix value and sends it to those nodes.
* The client, or a node acting on its behalf, then repeatedly queries nodes in that stream for the pubkey in question until it is retrieved.

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's prefix.
* Sets the remaining bits of the msg prefix nonce randomly.
* Sends the msg to one or more nodes with a stream number matching the destination address’s identifying prefix bits or a higher steam.

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* For each address owned by the client, the client connects to one or more nodes in the address's stream or a higher stream of the same branch.
* The client then requests any objects with a prefix nonce where the first n bits match the destination address's stream number, where n is the prefix length value of the address.
* The client then processes these objects.

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs).
When a client wants to subscribe to broadcasts from an address, it does the following:
* Extracts the address's prefix value from the address.
* Retrieves the address's pubkey (see above).
* Extracts the prefix length value from the address's pubkey and uses it to calculate the address's current stream number.
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address that the client is subscribed to.

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 (shown as "stream _" in the stream hierarchy diagram) will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 4 would mean that the node will deal with 6.25% of the total network object traffic.
* A prefix length value of 8 would mean that the node will deal with 0.39% of the total network object traffic.
* A prefix length value of 14 would mean that the node will deal with 0.006% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, each node would keep a large list of the addresses of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. The IP addresses and stream numbers of new nodes would be spread widely through the network, regardless of the stream system.

Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream.

Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How can this be done in a way that does not undermine the security of the network and its users?

The basic idea is that nodes can measure the level of object traffic in their stream. All objects must have POW done for them, so the object traffic is valuable because it is costly to spoof. Nodes would move to lower streams when the level of object traffic in their stream is too great for them to handle and move to higher streams when the level of object traffic is below 50% of what they are capable of processing.

In order to prevent attackers 'pushing' nodes into lower streams by creating short bursts of very high volume object traffic, nodes could implement a time delay mechanism that ensures that they will only move up or down based on the object traffic levels of a relatively long period of time (e.g. 1 week).

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How can this be done in a way that does not undermine the security of the network and its users?

Similarly to nodes, addresses must move to higher or lower streams in order to preserve the balance between anonymity and efficiency that the address's owner desires. Again, it would make sense to use the level of object traffic to decide when to do this. As clients must download all objects in their address's stream(s), they should also be able to use the levels of object traffic to determine when each address needs to move up or down in the stream hierarchy.

Clients that control addresses could also implement a time delay mechanism, similar to the one described above for nodes.

Scalability through Prefix Filtering

2015-02-03T13:08:50Z

JonathanCoe: Made some corrections

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network also has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* As the network grows or shrinks, nodes also move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* Nodes maintain long-lived connections to nodes in their own stream and streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines which streams the object will propagate in.
* Nodes process objects in their own stream and all lower streams of the same branch.
* Objects propagate in the stream that matches their prefix nonce and all higher streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals. In general the problems shared by most stream systems are simplified. For example, instead the network having to create or abandon streams as the total network traffic level changes, nodes simply adjust their position in a pre-determined hierarchy of streams, moving either 'up' or 'down' the hierarchy.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion possible 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing up to 100,000 objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects per day then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

===Proposed Stream Structure===

The diagram below shows the structure of streams under the proposed system. The small, light blue circles represent groups of nodes. The large coloured circles represent streams. Note that this diagram only shows 4 complete levels (5 levels of nodes and 4 levels of streams). The proposed stream hierarchy would have 64 such levels.

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

===Node Connections===

The diagram below illustrates the connections between nodes and clients under the proposed system. Both nodes and clients are labelled with a stream number, which correspond to the stream numbers in the stream structure diagram above. Note that this diagram only shows nodes and clients in the first 4 levels of streams. In practice there would be 64 levels of streams.

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Client Procedures ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a higher stream of the same branch.
* The pubkey propagates through its address's stream and all higher streams of the same branch.

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client, or a node acting on behalf of the client, connects to one or more nodes with a stream number that matches any number of bits of the address's prefix value.
* The client then creates a getpubkey object with a prefix nonce that exactly matches the address's prefix value and sends it to those nodes.
* The client, or a node acting on its behalf, then repeatedly queries nodes in that stream for the pubkey in question until it is retrieved.

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's prefix.
* Sets the remaining bits of the msg prefix nonce randomly.
* Sends the msg to one or more nodes with a stream number matching the destination address’s identifying prefix bits or a higher steam.

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* For each address owned by the client, the client connects to one or more nodes in the address's stream or a higher stream of the same branch.
* The client then requests any objects with a prefix nonce where the first n bits match the destination address's stream number, where n is the prefix length value of the address.
* The client then processes these objects.

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs).
When a client wants to subscribe to broadcasts from an address, it does the following:
* Extracts the address's prefix value from the address.
* Retrieves the address's pubkey (see above).
* Extracts the prefix length value from the address's pubkey and uses it to calculate the address's current stream number.
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address that the client is subscribed to.

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

The basic idea is that nodes can measure the level of object traffic in their stream. All objects must have POW done for them, so the object traffic is valuable because it is costly to spoof. Nodes would move to lower streams when the level of object traffic in their stream is too great for them to handle and move to higher streams when the level of object traffic is below 50% of what they are capable of processing.

In order to prevent attackers 'pushing' nodes into lower streams by creating short bursts of very high volume object traffic, nodes could implement a time delay mechanism that ensures that they will only move up or down based on the object traffic levels of a relatively long period of time (e.g. 1 week).

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Similarly to nodes, addresses must move to higher or lower streams in order to preserve the balance between anonymity and efficiency that the address's owner desires. Again, it would make sense to use the level of object traffic to decide when to do this. As clients must download all objects in their address's stream(s), they should also be able to use the levels of object traffic to determine when each address needs to move up or down in the stream hierarchy.

Clients that control addresses could also implement a time delay mechanism, similar to the one described above for nodes.

Scalability through Prefix Filtering

2015-02-03T12:58:24Z

JonathanCoe: Made some corrections

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network also has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* As the network grows or shrinks, nodes also move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* Nodes maintain long-lived connections to nodes in their own stream and streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines which streams the object will propagate in.
* Nodes process objects in their own stream and all lower streams of the same branch.
* Objects propagate in the stream that matches their prefix nonce and all higher streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals. In general the problems shared by most stream systems are simplified. For example, instead the network having to create or abandon streams as the total network traffic level changes, nodes simply adjust their position in a pre-determined hierarchy of streams, moving either 'up' or 'down' the hierarchy.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion possible 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing up to 100,000 objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects per day then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

===Proposed Stream Structure===

The diagram below shows the structure of streams under the proposed system. The small, light blue circles represent groups of nodes. The large coloured circles represent streams. Note that this diagram only shows 4 complete levels (5 levels of nodes and 4 levels of streams). The proposed stream hierarchy would have 64 such levels.

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

===Node Connections===

The diagram below illustrates the connections between nodes and clients under the proposed system. Both nodes and clients are labelled with a stream number, which correspond to the stream numbers in the stream structure diagram above. Note that this diagram only shows nodes and clients in the first 4 levels of streams. In practice there would be 64 levels of streams.

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Client Procedures ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a higher stream of the same branch.
* The pubkey propagates through its address's stream and all lower streams.

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client, or a node acting on behalf of the client, connects to one or more nodes with a stream number that matches the first n bits of the address's prefix value.
* The client then creates a getpubkey object with a prefix nonce that exactly matches the address's prefix value and sends it to those nodes.
* The client, or a node acting on its behalf, then then repeatedly queries nodes in that stream for the pubkey in question.

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey.
* Sets the remaining bits of the msg prefix nonce randomly.
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam.

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* For each address owned by the client, the client connects to one or more nodes in the address's stream or a higher stream of the same branch.
* The client then requests any objects with a prefix nonce where the first n bits match the destination address's stream number.
* The client then processes these objects.

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs).
When a client wants to subscribe to broadcasts from an address, it does the following:
* Extracts the address's prefix value from the address.
* Retrieves the address's pubkey (see above).
* Extracts the prefix length value from the address's pubkey and uses it to calculate the address's current stream number.
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address that the client is subscribed to.

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

The basic idea is that nodes can measure the level of object traffic in their stream. All objects must have POW done for them, so the object traffic is valuable because it is costly to spoof. Nodes would move to lower streams when the level of object traffic in their stream is too great for them to handle and move to higher streams when the level of object traffic is below 50% of what they are capable of processing.

In order to prevent attackers 'pushing' nodes into lower streams by creating short bursts of very high volume object traffic, nodes could implement a time delay mechanism that ensures that they will only move up or down based on the object traffic levels of a relatively long period of time (e.g. 1 week).

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Similarly to nodes, addresses must move to higher or lower streams in order to preserve the balance between anonymity and efficiency that the address's owner desires. Again, it would make sense to use the level of object traffic to decide when to do this. As clients must download all objects in their address's stream(s), they should also be able to use the levels of object traffic to determine when each address needs to move up or down in the stream hierarchy.

Clients that control addresses could also implement a time delay mechanism, similar to the one described above for nodes.

Scalability through Prefix Filtering

2015-02-03T12:43:55Z

JonathanCoe: Made some corrections

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network also has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* As the network grows or shrinks, nodes also move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* Nodes maintain long-lived connections to nodes in their own stream and streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines which streams the object will propagate in.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals. In general the problems shared by most stream systems are simplified. For example, instead the network having to create or abandon streams as the total network traffic level changes, nodes simply adjust their position in a pre-determined hierarchy of streams, moving either 'up' or 'down' the hierarchy.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion possible 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing up to 100,000 objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects per day then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

===Proposed Stream Structure===

The diagram below shows the structure of streams under the proposed system. The small, light blue circles represent groups of nodes. The large coloured circles represent streams. Note that this diagram only shows 4 complete levels (5 levels of nodes and 4 levels of streams). The proposed stream hierarchy would have 64 such levels.

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

===Node Connections===

The diagram below illustrates the connections between nodes and clients under the proposed system. Both nodes and clients are labelled with a stream number, which correspond to the stream numbers in the stream structure diagram above. Note that this diagram only shows nodes and clients in the first 4 levels of streams. In practice there would be 64 levels of streams.

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Client Procedures ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a lower stream of the same branch.
* The pubkey propagates through its address's stream and all lower streams.

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client, or a node acting on behalf of the client, connects to one or more nodes with a stream number that matches the first n bits of the address's prefix value.
* The client then creates a getpubkey object with a prefix nonce that exactly matches the address's prefix value and sends it to those nodes.
* The client, or a node acting on its behalf, then then repeatedly queries nodes in that stream for the pubkey in question.

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey.
* Sets the remaining bits of the msg prefix nonce randomly.
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam.

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* For each address owned by the client, the client connects to one or more nodes in the address's stream or a higher stream of the same branch.
* The client then requests any objects with a prefix nonce where the first n bits match the destination address's stream number.
* The client then processes these objects.

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs).
When a client wants to subscribe to broadcasts from an address, it does the following:
* Extracts the address's prefix value from the address.
* Retrieves the address's pubkey (see above).
* Extracts the prefix length value from the address's pubkey and uses it to calculate the address's current stream number.
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address that the client is subscribed to.

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

The basic idea is that nodes can measure the level of object traffic in their stream. All objects must have POW done for them, so the object traffic is valuable because it is costly to spoof. Nodes would move to lower streams when the level of object traffic in their stream is too great for them to handle and move to higher streams when the level of object traffic is below 50% of what they are capable of processing.

In order to prevent attackers 'pushing' nodes into lower streams by creating short bursts of very high volume object traffic, nodes could implement a time delay mechanism that ensures that they will only move up or down based on the object traffic levels of a relatively long period of time (e.g. 1 week).

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Similarly to nodes, addresses must move to higher or lower streams in order to preserve the balance between anonymity and efficiency that the address's owner desires. Again, it would make sense to use the level of object traffic to decide when to do this. As clients must download all objects in their address's stream(s), they should also be able to use the levels of object traffic to determine when each address needs to move up or down in the stream hierarchy.

Clients that control addresses could also implement a time delay mechanism, similar to the one described above for nodes.

Scalability through Prefix Filtering

2015-02-03T12:42:22Z

JonathanCoe: Added some content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network also has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* As the network grows or shrinks, nodes also move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* Nodes maintain long-lived connections to nodes in their own stream and streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines which streams the object will propagate in.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals. In general the problems shared by most stream systems are simplified. For example, instead the network having to create or abandon streams as the total network traffic level changes, nodes simply adjust their position in a pre-determined hierarchy of streams, moving either 'up' or 'down' the hierarchy.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion possible 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing 100,000 up to objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

===Proposed Stream Structure===

The diagram below shows the structure of streams under the proposed system. The small, light blue circles represent groups of nodes. The large coloured circles represent streams. Note that this diagram only shows 4 complete levels (5 levels of nodes and 4 levels of streams). The proposed stream hierarchy would have 64 such levels.

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

===Node Connections===

The diagram below illustrates the connections between nodes and clients under the proposed system. Both nodes and clients are labelled with a stream number, which correspond to the stream numbers in the stream structure diagram above. Note that this diagram only shows nodes and clients in the first 4 levels of streams. In practice there would be 64 levels of streams.

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Client Procedures ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a lower stream of the same branch.
* The pubkey propagates through its address's stream and all lower streams.

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client, or a node acting on behalf of the client, connects to one or more nodes with a stream number that matches the first n bits of the address's prefix value.
* The client then creates a getpubkey object with a prefix nonce that exactly matches the address's prefix value and sends it to those nodes.
* The client, or a node acting on its behalf, then then repeatedly queries nodes in that stream for the pubkey in question.

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey.
* Sets the remaining bits of the msg prefix nonce randomly.
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam.

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* For each address owned by the client, the client connects to one or more nodes in the address's stream or a higher stream of the same branch.
* The client then requests any objects with a prefix nonce where the first n bits match the destination address's stream number.
* The client then processes these objects.

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs).
When a client wants to subscribe to broadcasts from an address, it does the following:
* Extracts the address's prefix value from the address.
* Retrieves the address's pubkey (see above).
* Extracts the prefix length value from the address's pubkey and uses it to calculate the address's current stream number.
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address that the client is subscribed to.

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

The basic idea is that nodes can measure the level of object traffic in their stream. All objects must have POW done for them, so the object traffic is valuable because it is costly to spoof. Nodes would move to lower streams when the level of object traffic in their stream is too great for them to handle and move to higher streams when the level of object traffic is below 50% of what they are capable of processing.

In order to prevent attackers 'pushing' nodes into lower streams by creating short bursts of very high volume object traffic, nodes could implement a time delay mechanism that ensures that they will only move up or down based on the object traffic levels of a relatively long period of time (e.g. 1 week).

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Similarly to nodes, addresses must move to higher or lower streams in order to preserve the balance between anonymity and efficiency that the address's owner desires. Again, it would make sense to use the level of object traffic to decide when to do this. As clients must download all objects in their address's stream(s), they should also be able to use the levels of object traffic to determine when each address needs to move up or down in the stream hierarchy.

Clients that control addresses could also implement a time delay mechanism, similar to the one described above for nodes.

Scalability through Prefix Filtering

2015-02-03T12:09:22Z

JonathanCoe: Added some content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing 100,000 up to objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

===Proposed Stream Structure===

The diagram below shows the structure of streams under the proposed system. The small, light blue circles represent groups of nodes. The large coloured circles represent streams. Note that this diagram only shows 4 complete levels (5 levels of nodes and 4 levels of streams). The proposed stream hierarchy would have 64 such levels.

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

===Node Connections===

The diagram below illustrates the connections between nodes and clients under the proposed system. Both nodes and clients are labelled with a stream number, which correspond to the stream numbers in the stream structure diagram above. Note that this diagram only shows nodes and clients in the first 4 levels of streams. In practice there would be 64 levels of streams.

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Client Procedures ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a lower stream of the same branch.
* The pubkey propagates through its address's stream and all lower streams.

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client, or a node acting on behalf of the client, connects to one or more nodes with a stream number that matches the first n bits of the address's prefix value.
* The client then creates a getpubkey object with a prefix nonce that exactly matches the address's prefix value and sends it to those nodes.
* The client, or a node acting on its behalf, then then repeatedly queries nodes in that stream for the pubkey in question.

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey.
* Sets the remaining bits of the msg prefix nonce randomly.
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam.

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* For each address owned by the client, the client connects to one or more nodes in the address's stream or a higher stream of the same branch.
* The client then requests any objects with a prefix nonce where the first n bits match the destination address's stream number.
* The client then processes these objects.

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs).
When a client wants to subscribe to broadcasts from an address, it does the following:
* Extracts the address's prefix value from the address.
* Retrieves the address's pubkey (see above).
* Extracts the prefix length value from the address's pubkey and uses it to calculate the address's current stream number.
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address that the client is subscribed to.

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

The basic idea is that nodes can measure the level of object traffic in their stream. All objects must have POW done for them, so the object traffic is valuable because it is costly to spoof. Nodes would move to lower streams when the level of object traffic in their stream is too great for them to handle and move to higher streams when the level of object traffic is below 50% of what they are capable of processing.

In order to prevent attackers 'pushing' nodes into lower streams by creating short bursts of very high volume object traffic, nodes could implement a time delay mechanism that ensures that they will only move up or down based on the object traffic levels of a relatively long period of time (e.g. 1 week).

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Similarly to nodes, addresses must move to higher or lower streams in order to preserve the balance between anonymity and efficiency that the address's owner desires. Again, it would make sense to use the level of object traffic to decide when to do this. As clients must download all objects in their address's stream(s), they should also be able to use the levels of object traffic to determine when each address needs to move up or down in the stream hierarchy.

Clients that control addresses could also implement a time delay mechanism, similar to the one described above for nodes.

Scalability through Prefix Filtering

2015-02-02T18:38:46Z

JonathanCoe: Added some content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing 100,000 up to objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

===Proposed Stream Structure===

The diagram below shows the structure of streams under the proposed system. The small, light blue circles represent groups of nodes. The large coloured circles represent streams. Note that this diagram only shows 4 complete levels (5 levels of nodes and 4 levels of streams). The proposed stream hierarchy would have 64 such levels.

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

===Node Connections===

The diagram below illustrates the connections between nodes and clients under the proposed system. Both nodes and clients are labelled with a stream number, which correspond to the stream numbers in the stream structure diagram above. Note that this diagram only shows nodes and clients in the first 4 levels of streams. In practice there would be 64 levels of streams.

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Client Procedures ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a lower stream of the same branch.
* The pubkey propagates through its address's stream and all lower streams.

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client, or a node acting on behalf of the client, connects to one or more nodes with a stream number that matches the first n bits of the address's prefix value.
* The client then creates a getpubkey object with a prefix nonce that exactly matches the address's prefix value and sends it to those nodes.
* The client, or a node acting on its behalf, then then repeatedly queries nodes in that stream for the pubkey in question.

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey.
* Sets the remaining bits of the msg prefix nonce randomly.
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam.

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* For each address owned by the client, the client connects to one or more nodes in the address's stream or a higher stream of the same branch.
* The client then requests any objects with a prefix nonce where the first n bits match the destination address's stream number.
* The client then processes these objects.

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs).
When a client wants to subscribe to broadcasts from an address, it does the following:
* Extracts the address's prefix value from the address.
* Retrieves the address's pubkey (see above).
* Extracts the prefix length value from the address's pubkey and uses it to calculate the address's current stream number.
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address that the client is subscribed to.

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T18:36:03Z

JonathanCoe: Added some content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing 100,000 up to objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

===Proposed Stream Structure===

The diagram below shows the structure of streams under the proposed system. The small, light blue circles represent groups of nodes. The large coloured circles represent streams. Note that this diagram only shows 4 complete levels (5 levels of nodes and 4 levels of streams). The proposed stream hierarchy would have 64 such levels.

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

===Node Connections===

The diagram below illustrates the connections between nodes and clients under the proposed system. Both nodes and clients are labelled with a stream number, which correspond to the stream numbers in the stream structure diagram above. Note that this diagram only shows nodes and clients in the first 4 levels of streams. In practice there would be 64 levels of streams.

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Client Procedures ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a lower stream of the same branch.
* The pubkey propagates through its address's stream and all lower streams.

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client, or a node acting on behalf of the client, connects to one or more nodes with a stream number that matches the first n bits of the address's prefix value.
* The client then creates a getpubkey object for the required pubkey and sends it to those nodes.
* The client, or a node acting on its behalf, then then repeatedly queries nodes in that stream for the pubkey in question.

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey.
* Sets the remaining bits of the msg prefix nonce randomly.
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam.

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* For each address owned by the client, the client connects to one or more nodes in the address's stream or a higher stream of the same branch.
* The client then requests any objects with a prefix nonce where the first n bits match the destination address's stream number.
* The client then processes these objects.

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs).
When a client wants to subscribe to broadcasts from an address, it does the following:
* Extracts the address's prefix value from the address.
* Retrieves the address's pubkey (see above).
* Extracts the prefix length value from the address's pubkey and uses it to calculate the address's current stream number.
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address that the client is subscribed to.

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T18:25:30Z

JonathanCoe: Added some content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing 100,000 up to objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

===Proposed Stream Structure===

The diagram below shows the structure of streams under the proposed system. The small, light blue circles represent groups of nodes. The large coloured circles represent streams. Note that this diagram only shows 4 complete levels (5 levels of nodes and 4 levels of streams). The proposed stream hierarchy would have 64 such levels.

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

===Node Connections===

The diagram below illustrates the connections between nodes and clients under the proposed system. Both nodes and clients are labelled with a stream number, which correspond to the stream numbers in the stream structure diagram above. Note that this diagram only shows nodes and clients in the first 4 levels of streams. In practice there would be 64 levels of streams.

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Client Procedures ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a lower stream of the same branch.
* The pubkey propagates through its address's stream and all lower streams.

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client XXX

XXX How do you know when to create a getpubkey? You do not know how low the stream is... XXX

* The client creates a getpubkey object and sends it to the destination address's stream.
* The client, or a node acting on its behalf, then then repeatedly queries nodes in that stream for the pubkey in question.

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey.
* Sets the remaining bits of the msg prefix nonce randomly.
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam.

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* XXX

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs).
When a client wants to subscribe to broadcasts from an address, it does the following:
* Extracts the address's prefix value from the address.
* Retrieves the address's pubkey (see above).
* Extracts the prefix length value from the address's pubkey and uses it to calculate the address's current stream number.
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address that the client is subscribed to.

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T18:09:03Z

JonathanCoe: Added some content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing 100,000 up to objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

===Proposed Stream Structure===

The diagram below shows the structure of streams under the proposed system. The small, light blue circles represent groups of nodes. The large coloured circles represent streams. Note that this diagram only shows 4 complete levels (5 levels of nodes and 4 levels of streams). The proposed stream hierarchy would have 64 such levels.

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

===Node Connections===

The diagram below illustrates the connections between nodes and clients under the proposed system. Both nodes and clients are labelled with a stream number, which correspond to the stream numbers in the stream structure diagram above. Note that this diagram only shows nodes and clients in the first 4 levels of streams. In practice there would be 64 levels of streams.

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Examples ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a lower stream of the same branch.
* The pubkey propagates through its address's stream and all lower streams.

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client XXX

XXX How do you know when to create a getpubkey? You do not know how low the stream is... XXX

* The client creates a getpubkey object and sends it to the destination address's stream.
* The client, or a node acting on its behalf, then then repeatedly queries nodes in that stream for the pubkey in question.

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey.
* Sets the remaining bits of the msg prefix nonce randomly.
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam.

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* XXX

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs).
When a client wants to subscribe to broadcasts from an address, it does the following:
* Extracts the address's prefix value from the address.
* Retrieves the address's pubkey (see above).
* Extracts the prefix length value from the address's pubkey and uses it to calculate the address's current stream number.
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address that the client is subscribed to.

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T18:03:49Z

JonathanCoe: Added some content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing 100,000 up to objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

===Proposed Stream Structure===

The diagram below shows the structure of streams under the proposed system. The small, light blue circles represent groups of nodes. The large coloured circles represent streams. Note that this diagram only shows 4 complete levels (5 levels of nodes and 4 levels of streams). The proposed stream hierarchy would have 64 such levels.

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

===Node Connections===

The diagram below illustrates the connections between nodes and clients under the proposed system. Both nodes and clients are labelled with a stream number, which correspond to the stream numbers in the stream structure diagram above. Note that this diagram only shows nodes and clients in the first 4 levels of streams. In practice there would be 64 levels of streams.

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Examples ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a lower stream of the same branch
* The pubkey propagates through its address's stream and all lower streams

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client XXX

XXX How do you know when to create a getpubkey? You do not know how low the stream is... XXX

* Getpubkeys should be sent to the destination address's stream
* The retrieving node should then repeatedly query nodes in that stream for the pubkey in question

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey
* Sets the remaining bits of the msg prefix nonce randomly
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* XXX

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs)
If you want to subscribe to broadcasts from an address:
* Extract the address's prefix value from the address
* Retrieve the address's pubkey (see above)
* Extract the prefix length value from the address's pubkey and use it to calculate the address's current stream number
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address you are subscribed to

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T17:52:54Z

JonathanCoe: Added some content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing 100,000 up to objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Examples ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a lower stream of the same branch
* The pubkey propagates through its address's stream and all lower streams

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client XXX

XXX How do you know when to create a getpubkey? You do not know how low the stream is... XXX

* Getpubkeys should be sent to the destination address's stream
* The retrieving node should then repeatedly query nodes in that stream for the pubkey in question

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey
* Sets the remaining bits of the msg prefix nonce randomly
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* XXX

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs)
If you want to subscribe to broadcasts from an address:
* Extract the address's prefix value from the address
* Retrieve the address's pubkey (see above)
* Extract the prefix length value from the address's pubkey and use it to calculate the address's current stream number
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address you are subscribed to

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T17:51:37Z

JonathanCoe: Added some content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved (see https://bitmessage.org/wiki/Scalability_through_Prefix_Filtering#Unresolved_Questions).

Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing 100,000 up to objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

XXX Each address will have a 64 bit prefix value which is derivable from its address alone. Therefore a client with the address has all the information it needs to request the the pubkey with 100% efficiency or 100% anonymity.

XXX Every pubkey is guaranteed to be covered by many nodes, even if its owner has specified the most precise possible stream value. Therefore requests can be made to those nodes of whatever precision the requesting client desires.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Examples ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a lower stream of the same branch
* The pubkey propagates through its address's stream and all lower streams

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client XXX

XXX How do you know when to create a getpubkey? You do not know how low the stream is... XXX

* Getpubkeys should be sent to the destination address's stream
* The retrieving node should then repeatedly query nodes in that stream for the pubkey in question

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey
* Sets the remaining bits of the msg prefix nonce randomly
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* XXX

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs)
If you want to subscribe to broadcasts from an address:
* Extract the address's prefix value from the address
* Retrieve the address's pubkey (see above)
* Extract the prefix length value from the address's pubkey and use it to calculate the address's current stream number
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address you are subscribed to

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T17:44:18Z

JonathanCoe: Added some content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability. For example, if a node is capable of processing 100,000 up to objects per day and the total network object traffic is 10,000,000 objects per day then the node would select a prefix length value of 7, which would result in it having to process roughly 78,000 objects per day. Over time, the node can change its stream number in order to keep processing as much object traffic as it can.

When a Bitmessage address is created, the client creating the address selects a prefix length value that will result in a balance between anonymity and efficiency that best suits the end user who owns that address. For example, if the total network object traffic is 10,000,000 objects per day and the the owner of the new address wants a balance between anonymity and efficiency that equates to an anonymity set of roughly 10,000 objects then the client will select a prefix length value of 10 for the address. This will result in the client needing to download roughly 9,700 objects per day in order to be sure of receiving all objects destined for it.

XXX Each address will have a 64 bit prefix value which is derivable from its address alone. Therefore a client with the address has all the information it needs to request the the pubkey with 100% efficiency or 100% anonymity.

XXX Every pubkey is guaranteed to be covered by many nodes, even if its owner has specified the most precise possible stream value. Therefore requests can be made to those nodes of whatever precision the requesting client desires.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Examples ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a lower stream of the same branch
* The pubkey propagates through its address's stream and all lower streams

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client XXX

XXX How do you know when to create a getpubkey? You do not know how low the stream is... XXX

* Getpubkeys should be sent to the destination address's stream
* The retrieving node should then repeatedly query nodes in that stream for the pubkey in question

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey
* Sets the remaining bits of the msg prefix nonce randomly
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* XXX

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs)
If you want to subscribe to broadcasts from an address:
* Extract the address's prefix value from the address
* Retrieve the address's pubkey (see above)
* Extract the prefix length value from the address's pubkey and use it to calculate the address's current stream number
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address you are subscribed to

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T17:01:16Z

JonathanCoe: Added some content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Prefixes===
The purpose of the prefix filtering system is to provide a mechanism for determining how Bitmessage objects should be routed.

Rather than having one stream to start with and gradually increasing or decreasing the number of streams as the network grows or shrinks, this system makes 18 quintillion 'streams' (prefix values) available immediately.

Each node in the network handles a certain proportion of the total network traffic, with the size of that proportion determined according to the node's capability.

Each address XXX

XXX

XXX Each address will have a 64 bit prefix value which is derivable from its address alone. Therefore a client with the address has all the information it needs to request the the pubkey with 100% efficiency or 100% anonymity.

XXX Every pubkey is guaranteed to be covered by many nodes, even if its owner has specified the most precise possible stream value. Therefore requests can be made to those nodes of whatever precision the requesting client desires.

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

* Node addresses are shared very widely (e.g. up to 100 node addresses per stream).

* If a node needs to send an object to another stream, it connects to one or more nodes in that stream or a higher stream and sends the object to them.

* Nodes maintain many constant connections to nodes in their own stream and nearby streams (both higher and lower).

* Nodes of higher capacity should generally maintain more constant connections.

== Examples ==

===Creating a Bitmessage address and pubkey===
When a new Bitmessage address is generated by a Bitmessage client, the following occurs:
* The client makes a request to one or more nodes in the network for information on the current object traffic levels in the network.
* The client uses this information to determine a prefix length value that will result in a balance between anonymity and efficiency that matches the requirements of the owner of the address.
* The client creates a pubkey object for the address. This includes the prefix length value.
* The client takes the first n bits of the address's prefix value, where n is the prefix length value. The resulting value is the address's current stream number.
* The client sets the first n bits of pubkey prefix nonce to match the address's stream number.
* The client sets the remaining bits of the pubkey prefix nonce randomly.
* The client sends the pubkey to nodes in the address's stream or a lower stream of the same branch
* The pubkey propagates through its address's stream and all lower streams

===Retrieving an address's pubkey===
When a client needs to retrieve the pubkey of an address, it does the following:
* The client extracts the address's prefix value from the address string.
* The client XXX

XXX How do you know when to create a getpubkey? You do not know how low the stream is... XXX

* Getpubkeys should be sent to the destination address's stream
* The retrieving node should then repeatedly query nodes in that stream for the pubkey in question

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey
* Sets the remaining bits of the msg prefix nonce randomly
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* XXX

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs)
If you want to subscribe to broadcasts from an address:
* Extract the address's prefix value from the address
* Retrieve the address's pubkey (see above)
* Extract the prefix length value from the address's pubkey and use it to calculate the address's current stream number
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address you are subscribed to

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T16:28:54Z

JonathanCoe: Added some more content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Prefixes===
XXX

Rather than having 1 stream number to start with and gradually increasing the number of streams as the network grows, this system makes 18 quintillion 'streams' (prefix values) available immediately.

XXX

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

Node addresses are shared very widely (e.g. up to 100 node addresses per segment). It should be possible to detect and prevent the spread of fake node addresses through relatively simple testing methods.

If a node needs to send data to another segment, it needs to connect to a few nodes in that segment or an encompassing segment

Nodes should maintain many constant connections to nodes in their own segment and nearby segments (both higher and lower).
Nodes of higher capacity should generally maintain more constant connections.

== Examples ==

===Creating a Bitmessage address and pubkey===
* Client with address creates the pubkey object
* Client sets the first n bits of pubkey prefix nonce to match the prefix value of the address's stream
* Client sets the remaining bits of the pubkey prefix nonce randomly
* Client sends the pubkey to nodes in the address's stream or a lower stream of the same branch
* The pubkey propagates through its address's stream and all lower streams

===Retrieving an address's pubkey===
* Each address will have a 64 bit prefix value which is derivable from its address alone. Therefore a client with the address has all the information it needs to request the the pubkey with 100% efficiency or 100% anonymity.
* Every pubkey is guaranteed to be covered by many nodes, even if its owner has specified the most precise possible stream value. Therefore requests can be made to those nodes of whatever precision the requesting client desires.
* Getpubkeys should be sent to the destination address's stream
* The retrieving node should then repeatedly query nodes in that stream for the pubkey in question

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey
* Sets the remaining bits of the msg prefix nonce randomly
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* XXX

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs)
If you want to subscribe to broadcasts from an address:
* Extract the address's prefix value from the address
* Retrieve the address's pubkey (see above)
* Extract the prefix length value from the address's pubkey and use it to calculate the address's current stream number
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address you are subscribed to

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the node's stream number (which will be the same as its prefix value, since all 64 bits are used).

===Sharing of node addresses===
Under the proposed system, nodes would keep a very large list of other nodes, listing the IP address and stream number of each one. This list would cover a very large range of different stream numbers. Nodes would maintain long-lived connections to nodes in their own stream and 'nearby' streams in the tree hierarchy, but when necessary they could temporarily connect directly to nodes in any stream, for example when sending a message to an address in that stream. Nodes in higher streams would not have any special 'master' role. They would just handle a higher proportion of the object traffic than nodes in lower streams.

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T16:25:45Z

JonathanCoe: Added some more content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Prefixes===
XXX

Rather than having 1 stream number to start with and gradually increasing the number of streams as the network grows, this system makes 18 quintillion 'streams' (prefix values) available immediately.

XXX

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

Node addresses are shared very widely (e.g. up to 100 node addresses per segment). It should be possible to detect and prevent the spread of fake node addresses through relatively simple testing methods.

If a node needs to send data to another segment, it needs to connect to a few nodes in that segment or an encompassing segment

Nodes should maintain many constant connections to nodes in their own segment and nearby segments (both higher and lower).
Nodes of higher capacity should generally maintain more constant connections.

== Examples ==

===Creating a Bitmessage address and pubkey===
* Client with address creates the pubkey object
* Client sets the first n bits of pubkey prefix nonce to match the prefix value of the address's stream
* Client sets the remaining bits of the pubkey prefix nonce randomly
* Client sends the pubkey to nodes in the address's stream or a lower stream of the same branch
* The pubkey propagates through its address's stream and all lower streams

===Retrieving an address's pubkey===
* Each address will have a 64 bit prefix value which is derivable from its address alone. Therefore a client with the address has all the information it needs to request the the pubkey with 100% efficiency or 100% anonymity.
* Every pubkey is guaranteed to be covered by many nodes, even if its owner has specified the most precise possible stream value. Therefore requests can be made to those nodes of whatever precision the requesting client desires.
* Getpubkeys should be sent to the destination address's stream
* The retrieving node should then repeatedly query nodes in that stream for the pubkey in question

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey
* Sets the remaining bits of the msg prefix nonce randomly
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* XXX

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs)
If you want to subscribe to broadcasts from an address:
* Extract the address's prefix value from the address
* Retrieve the address's pubkey (see above)
* Extract the prefix length value from the address's pubkey and use it to calculate the address's current stream number
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address you are subscribed to

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached where no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or an even lower stream. This will mean that there will be no nodes in stream 1. This process of 'stream desertion' will continue as long as the object traffic continues to grow. However since both nodes and addresses can move to different streams, this should not present a problem for the functioning of the system.

===Node prefix length examples===
* A prefix length value of 0 would mean that the node will deal with 100% of the total network object traffic.
* A prefix length value of 5 would mean that the node will deal with 12.5% of the total network object traffic.
* A prefix length value of 64 would mean that the node will only deal with objects with a prefix nonce that exactly matches the nodes's stream number (which will be the same as its prefix value, since all 64 bits are used).

===XXX===
XXX

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T16:15:51Z

JonathanCoe: Added some more content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Prefixes===
XXX

Rather than having 1 stream number to start with and gradually increasing the number of streams as the network grows, this system makes 18 quintillion 'streams' (prefix values) available immediately.

XXX

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

Node addresses are shared very widely (e.g. up to 100 node addresses per segment). It should be possible to detect and prevent the spread of fake node addresses through relatively simple testing methods.

If a node needs to send data to another segment, it needs to connect to a few nodes in that segment or an encompassing segment

Nodes should maintain many constant connections to nodes in their own segment and nearby segments (both higher and lower).
Nodes of higher capacity should generally maintain more constant connections.

== Examples ==

===Creating a Bitmessage address and pubkey===
* Client with address creates the pubkey object
* Client sets the first n bits of pubkey prefix nonce to match the prefix value of the address's stream
* Client sets the remaining bits of the pubkey prefix nonce randomly
* Client sends the pubkey to nodes in the address's stream or a lower stream of the same branch
* The pubkey propagates through its address's stream and all lower streams

===Retrieving an address's pubkey===
* Each address will have a 64 bit prefix value which is derivable from its address alone. Therefore a client with the address has all the information it needs to request the the pubkey with 100% efficiency or 100% anonymity.
* Every pubkey is guaranteed to be covered by many nodes, even if its owner has specified the most precise possible stream value. Therefore requests can be made to those nodes of whatever precision the requesting client desires.
* Getpubkeys should be sent to the destination address's stream
* The retrieving node should then repeatedly query nodes in that stream for the pubkey in question

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey
* Sets the remaining bits of the msg prefix nonce randomly
* Sends the msg to nodes which handle the stream matching the destination address’s identifying prefix bits OR a higher steam

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* XXX

===Broadcasts===
Broadcasts are sent to every node in their destination address's stream (like msgs)
If you want to subscribe to broadcasts from an address:
* Extract the address's prefix value from the address
* Retrieve the address's pubkey (see above)
* Extract the prefix length value from the address's pubkey and use it to calculate the address's current stream number
* Periodically make a request to one or more nodes in the address's stream for any broadcast objects which are tagged as being from the address you are subscribed to

== Notes ==

===Upper Stream Desertion===
If the object traffic in the Bitmessage network continues to grow, eventually a point will be reached when no single node can process all of it. At this point, any nodes that started in stream 1 will have to move to stream 2 or 3, or a even lower stream. This will mean that there will be no nodes that XXX

===XXX===
XXX

===XXX===
XXX

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T15:57:13Z

JonathanCoe: Added some more content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

===Address===
The prefix value of each Bitmessage address is the first 64 bits of the ripe hash encoded within the address. The prefix of an address never changes.

The prefix length value of each address is contained within the address's pubkey, as below. The prefix length value of an address determines how many bits of the address's prefix should be used. Increasing the prefix length of an address moves that address to a lower stream. Increasing the prefix length of an address moves it to a higher stream. This can be accomplished by publishing a new pubkey which contains the updated prefix length value.

If non-hashed addresses are added to the Bitmessage protocol, their prefix length value will have to be encoded within the address string itself, as non-hashed addresses do not have pubkeys. This would prevent non-hashed addresses from moving between streams. Users of non-hashed addresses would have create new addresses instead. This would undoubtedly be less convenient, but is consistent with the overall profile of non-hashed addresses - gaining security and anonymity at the expense of convenience and efficiency.

===Object===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 8||prefixNonce||uint64_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

Node addresses are shared very widely (e.g. up to 100 node addresses per segment). It should be possible to detect and prevent the spread of fake node addresses through relatively simple testing methods.

If a node needs to send data to another segment, it needs to connect to a few nodes in that segment or an encompassing segment

Nodes should maintain many constant connections to nodes in their own segment and nearby segments (both higher and lower).
Nodes of higher capacity should generally maintain more constant connections.

== Examples ==

===Creating a Bitmessage address and pubkey===
* Client with address creates the pubkey object
* Client sets the first n bits of pubkey prefix nonce to match the prefix value of the address's stream
* Client sets the remaining bits of the pubkey prefix nonce randomly
* Client sends the pubkey to nodes in the address's stream or a lower stream of the same branch
* The pubkey propagates through its address's stream and all lower streams

===Retrieving an address's pubkey===
* Each address will have a 64 bit prefix value which is derivable from its address alone. Therefore a client with the address has all the information it needs to request the the pubkey with 100% efficiency or 100% anonymity.
* Every pubkey is guaranteed to be covered by many nodes, even if its owner has specified the most precise possible stream value. Therefore requests can be made to those nodes of whatever precision the requesting client desires.
* Getpubkeys should be sent to the destination addres's stream
* The retrieving node should then repeatedly query nodes in that stream for the pubkey in question

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey
* Sets the remaining bits of the msg prefix nonce randomly
* Sends the msg to nodes which handle the stream matching the desination addres's identifying prefix bits OR a higher steam

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T15:34:47Z

JonathanCoe: Added some more content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

=== Object ===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 4||prefixNonce||uint32_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

Node addresses are shared very widely (e.g. up to 100 node addresses per segment). It should be possible to detect and prevent the spread of fake node addresses through relatively simple testing methods.

If a node needs to send data to another segment, it needs to connect to a few nodes in that segment or an encompassing segment

Nodes should maintain many constant connections to nodes in their own segment and nearby segments (both higher and lower).
Nodes of higher capacity should generally maintain more constant connections.

== Examples ==

===Creating a Bitmessage address and pubkey===
* Client with address creates the pubkey object
* Client sets the first n bits of pubkey prefix nonce to match the prefix value of the address's stream
* Client sets the remaining bits of the pubkey prefix nonce randomly
* Client sends the pubkey to nodes in the address's stream or a lower stream of the same branch
* The pubkey propagates through its address's stream and all lower streams

===Retrieving an address's pubkey===
* Each address will have a 64 bit prefix value which is derivable from its address alone. Therefore a client with the address has all the information it needs to request the the pubkey with 100% efficiency or 100% anonymity.
* Every pubkey is guaranteed to be covered by many nodes, even if its owner has specified the most precise possible stream value. Therefore requests can be made to those nodes of whatever precision the requesting client desires.
* Getpubkeys should be sent to the destination addres's stream
* The retrieving node should then repeatedly query nodes in that stream for the pubkey in question

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey
* Sets the remaining bits of the msg prefix nonce randomly
* Sends the msg to nodes which handle the stream matching the desination addres's identifying prefix bits OR a higher steam

===Retrieving messages from the network===
When a client wishes to retrieve objects from the network, it does the following:
* XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T15:32:13Z

JonathanCoe: Added pubkey object table

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

=== Object ===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 4||prefixNonce||uint32_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

=== Pubkey ===
Under this proposal, the encrypted part of a pubkey would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the address's prefix value that should be used. Must be in the range 0-64.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

Node addresses are shared very widely (e.g. up to 100 node addresses per segment). It should be possible to detect and prevent the spread of fake node addresses through relatively simple testing methods.

If a node needs to send data to another segment, it needs to connect to a few nodes in that segment or an encompassing segment

Nodes should maintain many constant connections to nodes in their own segment and nearby segments (both higher and lower).
Nodes of higher capacity should generally maintain more constant connections.

== Examples ==

===Creating a Bitmessage address and pubkey===
* Client with address creates the pubkey object
* Client sets the first n bits of pubkey prefix nonce to match the prefix value of the address's stream
* Client sets the remaining bits of the pubkey prefix nonce randomly
* Client sends the pubkey to nodes in the address's stream or a lower stream of the same branch
* The pubkey propagates through its address's stream and all lower streams

===Retrieving an address's pubkey===
* Getpubkeys should be sent to the destination addres's stream
* The retrieving node should then repeatedly query nodes in that stream for the pubkey in question

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey
* Sets the remaining bits of the msg prefix nonce randomly
* Sends the msg to nodes which handle the stream matching the desination addres's identifying prefix bits OR a higher steam

===Retrieving messages from the network===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T15:29:23Z

JonathanCoe: Added some more content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

=== Object ===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 4||prefixNonce||uint32_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

===How do objects travel through the network / How do nodes connect to each other?===

Node addresses are shared very widely (e.g. up to 100 node addresses per segment). It should be possible to detect and prevent the spread of fake node addresses through relatively simple testing methods.

If a node needs to send data to another segment, it needs to connect to a few nodes in that segment or an encompassing segment

Nodes should maintain many constant connections to nodes in their own segment and nearby segments (both higher and lower).
Nodes of higher capacity should generally maintain more constant connections.

== Examples ==

===Creating a Bitmessage address and pubkey===
* Client with address creates the pubkey object
* Client sets the first n bits of pubkey prefix nonce to match the prefix value of the address's stream
* Client sets the remaining bits of the pubkey prefix nonce randomly
* Client sends the pubkey to nodes in the address's stream or a lower stream of the same branch
* The pubkey propagates through its address's stream and all lower streams

===Retrieving an address's pubkey===
* Getpubkeys should be sent to the destination addres's stream
* The retrieving node should then repeatedly query nodes in that stream for the pubkey in question

===Sending a message===
When a client sends a message to an address, it does the following:
* Sets the first n bits of msg prefix nonce to match the identifying prefix bits from the destination address's pubkey
* Sets the remaining bits of the msg prefix nonce randomly
* Sends the msg to nodes which handle the stream matching the desination addres's identifying prefix bits OR a higher steam

===Retrieving messages from the network===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T15:20:13Z

JonathanCoe:

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

=== Object ===

Under this proposal, Bitmessage objects would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 4||prefixNonce||uint32_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T15:19:24Z

JonathanCoe: Updates

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

=== Object ===

Under this proposal, a msg object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 4||prefixNonce||uint32_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T15:15:52Z

JonathanCoe:

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

====Nothing (or everyone gets everything)====
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

====Streams====
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

====Scaling without streams====
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

=== Object ===

Under this proposal, a msg object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 4||prefixNonce||uint32_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

===Idea: POW variable by prefix specificity===
Since

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T15:14:56Z

JonathanCoe:

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

Nothing (or everyone gets everything)
* Everyone user gets every message.
* Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
* Most private.

Streams
* Take the above and split it into pieces.
* Still potential for lots of bandwidth/disk space/processing usage.
* There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

Scaling without streams
* The same as the first method except only some messages are saved (once the network grows beyond a certain point).
* Requires two part messages.
* Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

=== Object ===

Under this proposal, a msg object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 4||prefixNonce||uint32_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

===Idea: POW variable by prefix specificity===
Since

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T15:13:20Z

JonathanCoe: Added some more content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

Nothing (or everyone gets everything)
- Everyone user gets every message.
- Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
- Most private.

Streams
- Take the above and split it into pieces.
- Still potential for lots of bandwidth/disk space/processing usage.
- There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

Scaling without streams
- The same as the first method except only some messages are saved (once the network grows beyond a certain point).
- Requires two part messages.
- Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

=== Object ===

Under this proposal, a msg object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 4||prefixNonce||uint32_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

XXX

===Proposed Stream Structure===

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

===Node Connections===

[[File:Node_Connections_Diagram.png|800px]]

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

===Idea: POW variable by prefix specificity===
Since

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T15:12:45Z

JonathanCoe: Added some more content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

Nothing (or everyone gets everything)
- Everyone user gets every message.
- Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
- Most private.

Streams
- Take the above and split it into pieces.
- Still potential for lots of bandwidth/disk space/processing usage.
- There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

Scaling without streams
- The same as the first method except only some messages are saved (once the network grows beyond a certain point).
- Requires two part messages.
- Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

=== Object ===

Under this proposal, a msg object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 4||prefixNonce||uint32_t||The object's prefix nonce. This determines which streams the object will propagate in.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

XXX

Proposed Stream Structure

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

Node Connections

[[File:Node_Connections_Diagram.png|800px]]

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

===Idea: POW variable by prefix specificity===
Since

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T15:10:18Z

JonathanCoe: Added object format table

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

Nothing (or everyone gets everything)
- Everyone user gets every message.
- Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
- Most private.

Streams
- Take the above and split it into pieces.
- Still potential for lots of bandwidth/disk space/processing usage.
- There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

Scaling without streams
- The same as the first method except only some messages are saved (once the network grows beyond a certain point).
- Requires two part messages.
- Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

=== object ===

Under this proposal, a msg object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#object.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 8||nonce||uint64_t||
Random nonce used for the [[Proof of work|Proof Of Work]]
|-
| 8||expiresTime||uint64_t||
The "end of life" time of this object (be aware, in version 2 of the protocol this was the generation time). Objects shall be shared with peers until its end-of-life time has been reached. The node should store the inventory vector of that object for some extra period of time to avoid reloading it from another node with a small time delay. The time may be no further than 28 days + 3 hours in the future.
|-
| 4||objectType||uint32_t||
Four values are currently defined: 0-"getpubkey", 1-"pubkey", 2-"msg", 3-"broadcast". All other values are reserved. Nodes should relay objects even if they use an undefined object type.
|-
| 1+||version||var_int||The object's version. Note that msg objects won't contain a version until Sun, 16 Nov 2014 22:00:00 GMT.
|-
| 4||prefixNonce||uint32_t||The prefix nonce.
|-
| ?||objectPayload||uchar[]||
This field varies depending on the object type; see below.
|-
|}

XXX

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

[[File:Node_Connections_Diagram.png|800px]]

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

===Idea: POW variable by prefix specificity===
Since

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T15:05:02Z

JonathanCoe: Added more content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values.
* As the network grows or shrinks, nodes move to a higher or lower stream in order to handle a greater or smaller proportion of the network object traffic.
* As the network grows or shrinks, addresses move to a higher or lower stream in order to maintain the balance between anonymity and efficiency desired by the address's owner.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* Each Bitmessage object has a prefix nonce. This value determines the object's destination stream.
* Objects propagate in their destination stream and lower streams of the same branch.
* Nodes process objects in their own stream and lower streams of the same branch.

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

Nothing (or everyone gets everything)
- Everyone user gets every message.
- Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
- Most private.

Streams
- Take the above and split it into pieces.
- Still potential for lots of bandwidth/disk space/processing usage.
- There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

Scaling without streams
- The same as the first method except only some messages are saved (once the network grows beyond a certain point).
- Requires two part messages.
- Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

XXX

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

[[File:Node_Connections_Diagram.png|800px]]

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

===Idea: POW variable by prefix specificity===
Since

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T14:58:48Z

JonathanCoe: Added summary of scalability possibilities

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values. As the network grows or shrinks, nodes change their stream in order to handle a smaller or greater proportion of the network objet traffic.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* XXX

== Reasoning behind the proposal ==

There are three basic possible approaches to making Bitmessage scalable (credit to Dokument for this summary):

Nothing (or everyone gets everything)
- Everyone user gets every message.
- Massive bandwidth and disk space and processing usage, eventually becomes completely unsustainable.
- Most private.

Streams
- Take the above and split it into pieces.
- Still potential for lots of bandwidth/disk space/processing usage.
- There are problems with binding addresses to one stream, and there are problems with not binding addresses to streams. Both sets affect privacy.

Scaling without streams
- The same as the first method except only some messages are saved (once the network grows beyond a certain point).
- Requires two part messages.
- Requires a lot of thought and processes to effectively hide receiving a message.

This proposal outlines a method for implementing streams that avoids or reduces many of the difficulties with previous stream proposals.

== Proposed changes ==

XXX

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

[[File:Node_Connections_Diagram.png|800px]]

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

===Idea: POW variable by prefix specificity===
Since

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T14:55:22Z

JonathanCoe: Added some more content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when retrieving messages from the network.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part and what proportion of the total network traffic the node will handle.
* Groups of nodes form overlapping 'streams' based on their prefix values. As the network grows or shrinks, nodes change their stream in order to handle a smaller or greater proportion of the network objet traffic.
* Nodes maintain long-lived connections to nodes in their own stream and 'nearby' streams that are 'nearby' in the stream hierarchy, but also temporarily connect directly to nodes in any stream when necessary.
* XXX

== Reasoning behind the proposal ==

XXX

== Proposed changes ==

XXX

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

[[File:Node_Connections_Diagram.png|800px]]

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

===Idea: POW variable by prefix specificity===
Since

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-02-02T11:52:04Z

JonathanCoe: Added diagrams

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when receiving messages.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part of the total network traffic the node will handle.
*
* XXX
* XXX

== Reasoning behind the proposal ==

XXX

== Proposed changes ==

XXX

[[File:Prefix_Filter_Streams_Hierarchy.png|800px]]

XXX

[[File:Node_Connections_Diagram.png|800px]]

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

===Idea: POW variable by prefix specificity===
Since

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

File:Node Connections Diagram.png

2015-02-02T11:47:32Z

JonathanCoe:

File:Prefix Filter Streams Hierarchy.png

2015-02-02T11:46:50Z

JonathanCoe:

Scalability through Prefix Filtering

2015-01-29T17:14:18Z

JonathanCoe: Added some more content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when receiving messages.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part of the total network traffic the node will handle.
*
* XXX
* XXX

== Reasoning behind the proposal ==

XXX

== Proposed changes ==

XXX

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

===Idea: POW variable by prefix specificity===
Since

== Unresolved Questions ==

===Rules for nodes moving between streams===
As the overall size of the network changes, nodes will need to adjust the proportion of the network traffic that they handle. This will require moving between streams. How should this be done?

===Rules for addresses moving between streams===
As the overall size of the network changes, addresses will need to move between streams in order to preserve the balance between anonymity and efficiency that their owner has selected. How should this be done?

Scalability through Prefix Filtering

2015-01-29T16:51:44Z

JonathanCoe: Started to fill out page content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

NOTE: This proposal is not yet complete, as some aspects of proposed system are not yet resolved. Suggestions and contributions are welcome.

== Summary of the proposal ==

* Each Bitmessage address has a 'prefix' and a 'prefix length'. These values determine the balance between anonymity and efficiency that the owner of the address will have when receiving messages.
* Each node in the Bitmessage network has a 'prefix' and a 'prefix length'. These values determine what part of the total network traffic the node will handle.
*
* XXX
* XXX

== Reasoning behind the proposal ==

XXX

== Proposed changes ==

XXX

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

===Idea: POW variable by prefix specificity===
Since

== Unresolved Questions ==

===Rules for nodes moving between streams===
XXX

===Rules for addresses moving between streams===
XXX

Scalability through Prefix Filtering

2015-01-20T16:25:39Z

JonathanCoe: Created outline of proposal page - still need to fill in content

{{Template:Proposal}}

== Introduction ==
This page describes a proposal for a way to make Bitmessage scalable.

== Summary of the proposal ==

* Each Bitmessage address has a unique 'prefix value' and a 'prefix length' value. These values determine the balance between anonymity and efficiency that the owner of the address will have when receiving XXX.
* Each node in the Bitmessage network chooses to deal with a certain proportion of the total network traffic, based on its capacity.
* Each then randomly selects a segment of the network with a size that matches the XXX
* XXX
* XXX

== Reasoning behind the proposal ==

XXX

== Proposed changes ==

XXX

== Examples ==

===Example 1===
XXX

===Example 2===
XXX

===Example 3===
XXX

== Notes ==

===XXX===
XXX

===XXX===
XXX

===XXX===
XXX

===Idea: POW variable by prefix specificity===
Since

Lite Client Message Retrieval Proposal

2015-01-04T16:13:41Z

JonathanCoe: /* Possible refinement: Daily changing prefix values */

{{Template:Proposal}}

== Introduction ==
This is a proposal to introduce a mechanism into the Bitmessage protocol that will allow 'lite' Bitmessage clients to retreive messages that have been sent to them without downloading all the messages in a stream.

== Summary of the proposal ==

* Lite clients can determine which messages they need to dowload using 'prefix filters'.
* Prefix filters allow the owner of each Bitmessage address to determine the balance between anonymity and efficiency that they will have when retrieving messages.
* Users who wish to retain the highest level of anonymity (by downloading all messages in a stream) are not affected by this change. Messages to them are 'tagged' with completely random data.
* Users who do not care about anonymity can specify a very precise prefix for each of their addresses, allowing them to download only messages destined for them.
* Other users can achieve a balance between anonymity and convenience.

== Reasoning behind the proposal ==

Currently, Bitmessage clients such as PyBitmessage act as full nodes in the Bitmessage peer-to-peer network. This involves transmitting, receiving, and processing a large amount of data. This workload may be manageable for more powerful devices, such as laptops, desktops, and servers, but it is typically too great for less powerful devices such as mobile phones and tablets to reasonably handle. This is particularly true in cases where mobile internet connections mean that bandwidth is very expensive. Because of this, a high proportion of the computing devices in use today are not suitable to act as full nodes in the Bitmessage peer-to-peer network. Therefore, in order for the users of these devices to be able to use Bitmessage, there must be some way of doing so that does not involve their device acting as a full node in the network.

One approach to solving this problem is the creation of 'lite' clients. These are Bitmessage clients which do not do all the processing and transmission work that full nodes do. Instead, the burden of this work is shifted to servers, which supply lite clients with the data they need and relay data sent by lite clients to the rest of the Bitmessage network. Thankfully the design of Bitmessage allows this to be implemented in such a way that very little security is lost. All message encryption and decryption can be done locally by the lite client, and it can retain sole control over its addresses and cryptographic keys. Data provided by servers can be cryptographically authenticated as being correct, and data sent to servers can be disseminated to several at once to ensure that it reaches the rest of the network.

However, this arrangement does require one significant compromise. In order for 'lite' clients such as mobile phones to receive messages, there needs to be some way for them to identify messages that have been sent to their addresses. Full nodes in Bitmessage do not require this because they receive and automatically attempt to decrypt all messages in the streams (segments of the network) which they are a part of. As described above however, the work required for this approach is too great for many devices, and therefore there must be some way for lite clients to identify which messages are destined for them, so that they can request those messages from a server.

== Proposed changes ==

The pubkey of every hashed address contains two new fields: a 'prefix length' field and a 'prefix' field. The value of the prefix length field is defined by the owner of the pubkey. The prefix field is set randomly. The value of the prefix length field defines what balance of efficiency and anonymity the owner of that address wants. If the value of the prefix length field is zero, then the prefix field can be ignored entirely, giving maximum anonymity. On the other extreme, if the value of the prefix length field is 32 (the maximum), then the full 32 bits of the prefix should be used, giving maximum efficiency in retrieving data. Each additional bit used halves the proportion of the total data set that you will have to request in order to receive data destined for you.

All msgs have a 'prefix nonce' as part of their unencrypted data. This prefix nonce is always of a fixed length (32 bits) to avoid leaking information. When sending a msg to an address, the sending client examines the destination address's pubkey and sets the first n bits of the prefix nonce to match the prefix found in the pubkey. The remainder of the prefix nonce is set randomly.

When a lite client wishes to retrieve msgs from the network, it makes a request to a server for all msgs which have a prefix nonce where the first n bits matches the prefix of the lite client's address(s).

=== Pubkey object ===
Under this proposal, the encrypted part of a pubkey object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the prefix value that should be used. Must be in the range 0-32.
|-
| 4||prefix||uint32_t||The prefix value. Contains 32 randomly set bits.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

=== Msg object ===
Under this proposal, a msg object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#msg.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 4||prefixNonce||uint32_t||The prefix nonce.
|-
| ?||encrypted||uchar[]||Encrypted data. See [[Protocol_specification#Encrypted_payload|Encrypted payload]]. See also [[Protocol_specification#Unencrypted_Message_Data|Unencrypted Message Data Format]]
|}

== Examples ==

===Example 1===
If Alice does not care about anonymity, she can have her address's pubkey specify that the full 32 bits of the prefix field should be used. This means that all msgs sent to her will effectively be uniquely tagged for her address (there are roughly 4 billion possible prefix values). This will allow Alice to only download msgs that are destined for her, giving her maximum usability and convenience at the expense of anonymity.

===Example 2===
If Bob wants to achieve a balance between anonymity and convenience, he can have his address's pubkey specify that 4 bits of the prefix field should be used, e.g 0110. This means that all msgs sent to his address will have a prefix nonce beginning with 0110. A random distribution of prefixes in a stream should mean that roughly 6.25% of msgs in a stream will have a prefix nonce beginning with 0110. Therefore Bob will be able to download roughly 6.25% of the msgs in a stream and still be guaranteed to receive all msgs destined for him.

===Example 3===
If Carol wants maximum anonymity then she can have her address's pubkey specify that zero bits of the prefix field should be used. This means that msgs sent to her will contain no identifying information, only a totally random prefix nonce. Carol will have to download and attempt to decrypt all msgs in a stream in order to be sure of receiving all msgs destined for her.

== Notes ==

===Other object types===
Prefix filtering is not required for broadcasts, pubkeys, or getpubkeys. These object types are already tagged with an identifier.

===Lite client pubkey retrieval===
How can lite clients retrieve pubkeys without telling the servers who they are talking to? 
In the short to medium term (until streams are implemented), it will be perfectly viable for lite clients to simply download all pubkeys in stream 1. A pubkey is 392 bytes in size. If we assume an average of 5 addresses per user, we could support a 10x increase in the number of active users (from roughly 1,000 to 10,000), and lite clients would still only have to download 19MB worth of pubkey data. This is only a rough approximation, but it seems reasonably clear that the amount of data to download and store would be acceptable.

===Lite client pubkey dissemination===
How can lite clients respond to getpubkey requests? 
Lite clients can do this in two ways: 
1. They can make periodic requests to full node servers to check for any getpubkeys for any of their addresses and then respond to any getpubkeys they receive by re-disseminating the relevant pubkeys. 
2. They can periodically re-disseminate the pubkeys for each of their addresses, with the re-dissemiantion period determined by the time to live of the pubkeys they create. This will ensure that the pubkeys for all the lite client's addresses are always available in the stream(s) they are a part of.

===Non-hashed addresses===
If non-hashed addresses are introduced in the future, this scheme will not apply to them, as they will not have pubkeys. Users of non-hashed addresses will still need to download and attempt to decrypt all msgs in each stream in which they have an address. This makes sense though, as the whole point of introducing non-hashed addresses is to give people the ability to choose greater anonymity at the expense of convenience.

===Streams===
The system described in this proposal is intended to be complementary to streams (i.e. both should be implemented). It is not intended to replace them.

===Possible refinement: Daily changing prefix values===
While the system described above gives a reasonable balance between anonymity and efficiency, it has some weaknesses. In particular, users sending messages to addresses which specify a non-zero prefix length value will tend to leak some information about who they are communicating with. By observing the prefix nonces of msgs sent by a particular client, an attacker may be able build up information about how many different addresses a given user is communicating with, who they are communicating with more often, and who they do not communicate with for certain periods of time.

One possible way to mitigate this problem would be to modify the prefix filtering system so that the prefix values used to 'tag' msgs vary according to a time value. Instead of tagging msgs with the first n bits of the prefix defined in the destination address's pubkey, msgs could be tagged with the first n bits of a hash value. The hash value would be calculated from the first n bits of the prefix concatenated with a time value. The result of this would be that the identifying bits of the prefix nonce of a msg would change every so often in a predictable way, with the frequency of change defined by the time value used.

For example, if the time value is defined by the protocol as 00:00 on the current day in Unix time, then the identifying prefix bits used to tag msgs will be different each day, in a way that can be easily and reliably calculated by both the sender and the receiver of msgs. The result of this is that rather than all msgs ever sent to a given address having the same identifying prefix bits, all msgs sent to a given address on a given day have the same identifying prefix bits.

A rough pseudocode description of the procedure to calculate the prefixNonce for a msg follows:
unixTime = getUnixTime()
remainderSeconds = unixTime mod 86,400
timeValue = unixTime - remainderSeconds
hash = sha512(sha512(prefix[0-prefixLength] + timeValue))
identifyingBits = hash[0-prefixLength]
prefixNonce = identifyingBits + randomBits

Advantages:
* The information leaked by a client sending msgs to addresses which specify a non-zero prefix length value is greatly diminished, as the identifying bits in the prefix nonces of sent msgs change each day. Some information is still leaked, particularly when an attacker already knows the destination address(es) of sent msgs and can therefore calculate what the identifying prefix bits will be each day, but the overall amount of information leaked is substantially reduced.

Disadvantages:
* When catching up with the network, lite clients must make a separate request for messages for each day in the time period that they need to check.

Lite Client Message Retrieval Proposal

2015-01-04T16:12:11Z

JonathanCoe: Added description of daily changing prefix values idea

{{Template:Proposal}}

== Introduction ==
This is a proposal to introduce a mechanism into the Bitmessage protocol that will allow 'lite' Bitmessage clients to retreive messages that have been sent to them without downloading all the messages in a stream.

== Summary of the proposal ==

* Lite clients can determine which messages they need to dowload using 'prefix filters'.
* Prefix filters allow the owner of each Bitmessage address to determine the balance between anonymity and efficiency that they will have when retrieving messages.
* Users who wish to retain the highest level of anonymity (by downloading all messages in a stream) are not affected by this change. Messages to them are 'tagged' with completely random data.
* Users who do not care about anonymity can specify a very precise prefix for each of their addresses, allowing them to download only messages destined for them.
* Other users can achieve a balance between anonymity and convenience.

== Reasoning behind the proposal ==

Currently, Bitmessage clients such as PyBitmessage act as full nodes in the Bitmessage peer-to-peer network. This involves transmitting, receiving, and processing a large amount of data. This workload may be manageable for more powerful devices, such as laptops, desktops, and servers, but it is typically too great for less powerful devices such as mobile phones and tablets to reasonably handle. This is particularly true in cases where mobile internet connections mean that bandwidth is very expensive. Because of this, a high proportion of the computing devices in use today are not suitable to act as full nodes in the Bitmessage peer-to-peer network. Therefore, in order for the users of these devices to be able to use Bitmessage, there must be some way of doing so that does not involve their device acting as a full node in the network.

One approach to solving this problem is the creation of 'lite' clients. These are Bitmessage clients which do not do all the processing and transmission work that full nodes do. Instead, the burden of this work is shifted to servers, which supply lite clients with the data they need and relay data sent by lite clients to the rest of the Bitmessage network. Thankfully the design of Bitmessage allows this to be implemented in such a way that very little security is lost. All message encryption and decryption can be done locally by the lite client, and it can retain sole control over its addresses and cryptographic keys. Data provided by servers can be cryptographically authenticated as being correct, and data sent to servers can be disseminated to several at once to ensure that it reaches the rest of the network.

However, this arrangement does require one significant compromise. In order for 'lite' clients such as mobile phones to receive messages, there needs to be some way for them to identify messages that have been sent to their addresses. Full nodes in Bitmessage do not require this because they receive and automatically attempt to decrypt all messages in the streams (segments of the network) which they are a part of. As described above however, the work required for this approach is too great for many devices, and therefore there must be some way for lite clients to identify which messages are destined for them, so that they can request those messages from a server.

== Proposed changes ==

The pubkey of every hashed address contains two new fields: a 'prefix length' field and a 'prefix' field. The value of the prefix length field is defined by the owner of the pubkey. The prefix field is set randomly. The value of the prefix length field defines what balance of efficiency and anonymity the owner of that address wants. If the value of the prefix length field is zero, then the prefix field can be ignored entirely, giving maximum anonymity. On the other extreme, if the value of the prefix length field is 32 (the maximum), then the full 32 bits of the prefix should be used, giving maximum efficiency in retrieving data. Each additional bit used halves the proportion of the total data set that you will have to request in order to receive data destined for you.

All msgs have a 'prefix nonce' as part of their unencrypted data. This prefix nonce is always of a fixed length (32 bits) to avoid leaking information. When sending a msg to an address, the sending client examines the destination address's pubkey and sets the first n bits of the prefix nonce to match the prefix found in the pubkey. The remainder of the prefix nonce is set randomly.

When a lite client wishes to retrieve msgs from the network, it makes a request to a server for all msgs which have a prefix nonce where the first n bits matches the prefix of the lite client's address(s).

=== Pubkey object ===
Under this proposal, the encrypted part of a pubkey object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 1||prefix_length||uint8_t||The number of bits from the prefix value that should be used. Must be in the range 0-32.
|-
| 4||prefix||uint32_t||The prefix value. Contains 32 randomly set bits.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

=== Msg object ===
Under this proposal, a msg object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#msg.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 4||prefixNonce||uint32_t||The prefix nonce.
|-
| ?||encrypted||uchar[]||Encrypted data. See [[Protocol_specification#Encrypted_payload|Encrypted payload]]. See also [[Protocol_specification#Unencrypted_Message_Data|Unencrypted Message Data Format]]
|}

== Examples ==

===Example 1===
If Alice does not care about anonymity, she can have her address's pubkey specify that the full 32 bits of the prefix field should be used. This means that all msgs sent to her will effectively be uniquely tagged for her address (there are roughly 4 billion possible prefix values). This will allow Alice to only download msgs that are destined for her, giving her maximum usability and convenience at the expense of anonymity.

===Example 2===
If Bob wants to achieve a balance between anonymity and convenience, he can have his address's pubkey specify that 4 bits of the prefix field should be used, e.g 0110. This means that all msgs sent to his address will have a prefix nonce beginning with 0110. A random distribution of prefixes in a stream should mean that roughly 6.25% of msgs in a stream will have a prefix nonce beginning with 0110. Therefore Bob will be able to download roughly 6.25% of the msgs in a stream and still be guaranteed to receive all msgs destined for him.

===Example 3===
If Carol wants maximum anonymity then she can have her address's pubkey specify that zero bits of the prefix field should be used. This means that msgs sent to her will contain no identifying information, only a totally random prefix nonce. Carol will have to download and attempt to decrypt all msgs in a stream in order to be sure of receiving all msgs destined for her.

== Notes ==

===Other object types===
Prefix filtering is not required for broadcasts, pubkeys, or getpubkeys. These object types are already tagged with an identifier.

===Lite client pubkey retrieval===
How can lite clients retrieve pubkeys without telling the servers who they are talking to? 
In the short to medium term (until streams are implemented), it will be perfectly viable for lite clients to simply download all pubkeys in stream 1. A pubkey is 392 bytes in size. If we assume an average of 5 addresses per user, we could support a 10x increase in the number of active users (from roughly 1,000 to 10,000), and lite clients would still only have to download 19MB worth of pubkey data. This is only a rough approximation, but it seems reasonably clear that the amount of data to download and store would be acceptable.

===Lite client pubkey dissemination===
How can lite clients respond to getpubkey requests? 
Lite clients can do this in two ways: 
1. They can make periodic requests to full node servers to check for any getpubkeys for any of their addresses and then respond to any getpubkeys they receive by re-disseminating the relevant pubkeys. 
2. They can periodically re-disseminate the pubkeys for each of their addresses, with the re-dissemiantion period determined by the time to live of the pubkeys they create. This will ensure that the pubkeys for all the lite client's addresses are always available in the stream(s) they are a part of.

===Non-hashed addresses===
If non-hashed addresses are introduced in the future, this scheme will not apply to them, as they will not have pubkeys. Users of non-hashed addresses will still need to download and attempt to decrypt all msgs in each stream in which they have an address. This makes sense though, as the whole point of introducing non-hashed addresses is to give people the ability to choose greater anonymity at the expense of convenience.

===Streams===
The system described in this proposal is intended to be complementary to streams (i.e. both should be implemented). It is not intended to replace them.

===Possible refinement: Daily changing prefix values===
While the system described above gives a reasonable balance between anonymity and efficiency, it has some weaknesses. In particular, users sending messages to addresses which specify a non-zero prefix length value will tend to leak some information about who they are communicating with. By observing the prefix nonces of msgs sent by a particular node, an attacker may be able build up information about how many different addresses a given user is communicating with, who they are communicating with more often, and who they do not communicate with for certain periods of time.

One possible way to mitigate this problem would be to modify the prefix filtering system so that the prefix values used to 'tag' msgs vary according to a time value. Instead of tagging msgs with the first n bits of the prefix defined in the destination address's pubkey, msgs could be tagged with the first n bits of a hash value. The hash value would be calculated from the first n bits of the prefix concatenated with a time value. The result of this would be that the identifying bits of the prefix nonce of a msg would change every so often in a predictable way, with the frequency of change defined by the time value used.

For example, if the time value is defined by the protocol as 00:00 on the current day in Unix time, then the identifying prefix bits used to tag msgs will be different each day, in a way that can be easily and reliably calculated by both the sender and the receiver of msgs. The result of this is that rather than all msgs ever sent to a given address having the same identifying prefix bits, all msgs sent to a given address on a given day have the same identifying prefix bits.

A rough pseudocode description of the procedure to calculate the prefixNonce for a msg follows:
unixTime = getUnixTime()
remainderSeconds = unixTime mod 86,400
timeValue = unixTime - remainderSeconds
hash = sha512(sha512(prefix[0-prefixLength] + timeValue))
identifyingBits = hash[0-prefixLength]
prefixNonce = identifyingBits + randomBits

Advantages:
* The information leaked by a user sending msgs to addresses which specify a non-zero prefix length value is greatly diminished, as the identifying bits in the prefix nonces of sent msgs change each day. Some information is still leaked, particularly when an attacker already knows the destination address(es) of sent msgs and can therefore calculate what the identifying prefix bits will be each day, but the overall amount of information leaked is substantially reduced.

Disadvantages:
* When catching up with the network, lite clients must make a separate request for messages for each day in the time period that they need to check.

Lite Client Message Retrieval Proposal

2015-01-04T14:35:15Z

JonathanCoe: Added section for 'Daily changing prefix values' idea

Lite Client Message Retrieval Proposal

2015-01-04T14:12:39Z

JonathanCoe: Reverted to using prefix length instead of prefix mask

Developer Reference

2014-12-23T15:06:56Z

JonathanCoe: Added bitmessage-ruby

This page is intended to provide information and other resources that are useful for Bitmessage developers.

==Protocol Information==

Protocol specification: https://bitmessage.org/wiki/Protocol_specification

Encryption scheme: https://bitmessage.org/wiki/Encryption

Proof of Work scheme: https://bitmessage.org/wiki/Proof_of_work

Bitmessage White Paper: https://bitmessage.org/bitmessage.pdf

Bitmessage Technical Paper (Note: not updated for protocol version 3): https://bitmessage.org/Bitmessage%20Technical%20Paper.pdf

== Implementations ==

===Full Node Implementations===

PyBitmessage (Reference Client) (Python): https://github.com/bitmessage/pybitmessage

bitmessaged (C++): https://github.com/Thomas-Astade/bitmessaged

===Lite Client Implementations===

Bitseal (Java): https://github.com/jonathancoe/bitseal

Notbit (C): https://github.com/bpeel/notbit

Bmr (Javascript): https://github.com/chovy/bmr

bitmessage-web (Javascript): https://github.com/jclement/bitmessage-web

Bitpost (Objective-C): https://github.com/Voluntarynet/Bitpost

===Web Clients===

Blinked (Javascript): https://blinked.ca

Bitmsg.me (Javascript): https://bitmsg.me

===Gateway Services===

https://bitmessage.ch

http://bitmessage.mobi

===Other Implementations===

Please note that some of these other implementations may be incomplete or not up-to-date with the current Bitmessage protocol.

libbitmessage (C++): https://github.com/corebob/libbitmessage

bitmessage-go (Go): https://github.com/corebob/bitmessage-go

cppbitmessage (C++): https://github.com/bashrc/cppbitmessage

JBitmessage (Java): https://github.com/ISibboI/JBitmessage

SharpBitmessage (C#): https://github.com/sharpbitmessage/SharpBitmessage

Bitmessage-js (Javascript): https://github.com/indigots/Bitmessage-js

bitmessage-ruby (Ruby): https://github.com/staii/bitmessage-ruby

==Scripts and Utilities==

===PyBitmessage Utilities===

Bitmessage PHP class - Bitmessage PHP Class to control PyBitmessage daemon using xmlrpc - https://github.com/Conver/class.bitmessage.php

bmwrapper - Email wrapper for PyBitmessage: https://github.com/Arceliar/bmwrapper

BitCrypt - Encrypts and decrypts PyBitmessage .dat files: https://github.com/AyrA/BitCrypt

BitUpdate - Automatically update PyBitmessage: https://github.com/AyrA/BitUpdate

===Others===

bitmessage-powfaster - Bitmessage Proof Of Work optimizations including OpenCL and C based PoW code: https://github.com/grant-olson/bitmessage-powfaster

BitMailServer - Bitmesssage to Email Gateway: https://github.com/AyrA/BitMailServer

BitDNS - Bitmessage DNS and Namecoin integration: https://github.com/AyrA/BitDNS

BitCenter - Powerful Bitmessage message processing: https://github.com/AyrA/BitCenter

BitHTTP - HTTP proxy over Bitmessage: https://github.com/AyrA/BitHTTP

BinSend - Send and decode binary attachments via Bitmessage: https://github.com/AyrA/BinSend

BitMessageForum - Browse your bitmessages via a forum-like UI: https://github.com/grant-olson/BitMessageForum

Lite Client Message Retrieval Proposal

2014-12-23T11:08:58Z

JonathanCoe: Minor formatting correction

{{Template:Proposal}}

== Introduction ==
This is a proposal to introduce a mechanism into the Bitmessage protocol that will allow 'lite' Bitmessage clients to retreive messages that have been sent to them without downloading all the messages in a stream.

== Summary of the proposal ==

* Lite clients can determine which messages they need to dowload using 'prefix filters'.
* Prefix filters allow the owner of each Bitmessage address to determine the balance between anonymity and efficiency that they will have when retrieving messages.
* Users who wish to retain the highest level of anonymity (by downloading all messages in a stream) are not affected by this change. Messages to them are 'tagged' with completely random data.
* Users who do not care about anonymity can specify a very precise prefix for each of their addresses, allowing them to download only messages destined for them.
* Other users can achieve a balance between anonymity and convenience.

== Reasoning behind the proposal ==

Currently, Bitmessage clients such as PyBitmessage act as full nodes in the Bitmessage peer-to-peer network. This involves transmitting, receiving, and processing a large amount of data. This workload may be manageable for more powerful devices, such as laptops, desktops, and servers, but it is typically too great for less powerful devices such as mobile phones and tablets to reasonably handle. This is particularly true in cases where mobile internet connections mean that bandwidth is very expensive. Because of this, a high proportion of the computing devices in use today are not suitable to act as full nodes in the Bitmessage peer-to-peer network. Therefore, in order for the users of these devices to be able to use Bitmessage, there must be some way of doing so that does not involve their device acting as a full node in the network.

One approach to solving this problem is the creation of 'lite' clients. These are Bitmessage clients which do not do all the processing and transmission work that full nodes do. Instead, the burden of this work is shifted to servers, which supply lite clients with the data they need and relay data sent by lite clients to the rest of the Bitmessage network. Thankfully the design of Bitmessage allows this to be implemented in such a way that very little security is lost. All message encryption and decryption can be done locally by the lite client, and it can retain sole control over its addresses and cryptographic keys. Data provided by servers can be cryptographically authenticated as being correct, and data sent to servers can be disseminated to several at once to ensure that it reaches the rest of the network.

However, this arrangement does require one significant compromise. In order for 'lite' clients such as mobile phones to receive messages, there needs to be some way for them to identify messages that have been sent to their addresses. Full nodes in Bitmessage do not require this because they receive and automatically attempt to decrypt all messages in the streams (segments of the network) which they are a part of. As described above however, the work required for this approach is too great for many devices, and therefore there must be some way for lite clients to identify which messages are destined for them, so that they can request those messages from a server.

== Proposed changes ==

The pubkey of every hashed address contains two new fields: a 'prefix mask' field and a 'prefix' field. The value of the prefix mask field is determined by the owner of the pubkey. The prefix field is set randomly. The value of the prefix mask field defines what balance of efficiency and anonymity the owner of that address wants. If the owner of the pubkey wants maximum anonymity, then all the bits in the prefix mask field will be set to 0, allowing the prefix to be ignored entirely. On the other extreme, if the owner of the pubkey wants maximum efficiency in retrieving data at the expense of anonymity then all bits in the prefix mask field will be set to 1. Each additional bit set to one halves the proportion of the total data set that you will have to request in order to receive data destined for you. The positions of the bits set to 1 in the prefix mask field are selected randomly.

All msgs have a 'prefix nonce' as part of their unencrypted data. This prefix nonce is always of a fixed length (32 bits) to avoid leaking information. When sending a msg to an address, the sending client examines the destination pubkey's prefix mask field and records which bit positions (if any) are set to 1. The sending client will then set the bits in the corresponding positions of the prefix nonce to match those found in the destination pubkey's prefix field. The remaining bits of the prefix nonce are set randomly.

When a lite client wishes to retrieve msgs from the network, it makes a request to a server for all msgs which have a prefix nonce where the bits set to 1 in the prefix mask of the receiving address's pubkey match the prefix field of the receiving pubkey.

=== Pubkey object ===
Under this proposal, the encrypted part of a pubkey object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 4||prefix_mask||uint32_t||A bit mask that defines which bits from the prefix field should be used.
|-
| 4||prefix||uint32_t||The prefix value. Contains 32 randomly set bits.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

=== Msg object ===
Under this proposal, a msg object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#msg.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 4||prefix_nonce||uint32_t||The prefix nonce.
|-
| ?||encrypted||uchar[]||Encrypted data. See [[Protocol_specification#Encrypted_payload|Encrypted payload]]. See also [[Protocol_specification#Unencrypted_Message_Data|Unencrypted Message Data Format]]
|}

== Examples ==

===Example 1===
If Alice does not care about anonymity, she can have her address's pubkey specify that the full 32 bits of the prefix field should be used. This is done by setting all bits of the pubkey's prefix mask field to 1. This means that all msgs sent to her will effectively be uniquely tagged for her address (there are roughly 4 billion possible prefix values). This will allow Alice to only download msgs that are destined for her, giving her maximum usability and convenience at the expense of anonymity.

===Example 2===
If Bob wants to achieve a balance between anonymity and convenience, he can have his address's pubkey specify that 4 bits of the prefix field should be used. This is done by setting 4 randomly selected bits of the pubkey's prefix mask field to 1. This means that all msgs sent to his address will have a prefix nonce where the 4 bits in the positions set to 1 in the prefix mask will match the corresponding bits in his pubkey's prefix field. A random distribution of prefixes in a stream should mean that roughly 6.25% of msgs in a stream will have a prefix nonce where the bits in those positions match the values of those bits in his pubkey's prefix field. Therefore Bob will be able to download roughly 6.25% of the msgs in a stream and still be guaranteed to receive all msgs destined for him.

===Example 3===
If Carol wants maximum anonymity then she can have her address's pubkey specify that zero bits of the prefix field should be used. This is done by setting all of the bits of the pubkey's prefix mask field to 0. This means that msgs sent to her will contain no identifying information, only a totally random prefix nonce. Carol will have to download and attempt to decrypt all msgs in a stream in order to be sure of receiving all msgs destined for her.

== Notes ==

===Other object types===
Prefix filtering is not required for broadcasts, pubkeys, or getpubkeys. These object types are already tagged with an identifier.

===Lite client pubkey retrieval===
How can lite clients retrieve pubkeys without telling the servers who they are talking to? 
In the short to medium term (until streams are implemented), it will be perfectly viable for lite clients to simply download all pubkeys in stream 1. A pubkey is 392 bytes in size. If we assume an average of 5 addresses per user, we could support a 10x increase in the number of active users (from roughly 1,000 to 10,000), and lite clients would still only have to download 19MB worth of pubkey data. This is only a rough approximation, but it seems reasonably clear that the amount of data to download and store would be acceptable.

===Lite client pubkey dissemination===
How can lite clients respond to getpubkey requests? 
Lite clients can do this in two ways: 
1. They can make periodic requests to full node servers to check for any getpubkeys for any of their addresses and then respond to any getpubkeys they receive by re-disseminating the relevant pubkeys. 
2. They can periodically re-disseminate the pubkeys for each of their addresses, with the re-dissemiantion period determined by the time to live of the pubkeys they create. This will ensure that the pubkeys for all the lite client's addresses are always available in the stream(s) they are a part of.

===Non-hashed addresses===
If non-hashed addresses are introduced in the future, this scheme will not apply to them, as they will not have pubkeys. Users of non-hashed addresses will still need to download and attempt to decrypt all msgs in each stream in which they have an address. This makes sense though, as the whole point of introducing non-hashed addresses is to give people the ability to choose greater anonymity at the expense of convenience.

===Streams===
The system described in this proposal is intended to be complementary to streams (i.e. both should be implemented). It is not intended to replace them.

Lite Client Message Retrieval Proposal

2014-12-23T10:58:31Z

JonathanCoe:

{{Template:Proposal}}

== Introduction ==
This is a proposal to introduce a mechanism into the Bitmessage protocol that will allow 'lite' Bitmessage clients to retreive messages that have been sent to them without downloading all the messages in a stream.

== Summary of the proposal ==

* Lite clients can determine which messages they need to dowload using 'prefix filters'.
* Prefix filters allow the owner of each Bitmessage address to determine the balance between anonymity and efficiency that they will have when retrieving messages.
* Users who wish to retain the highest level of anonymity (by downloading all messages in a stream) are not affected by this change. Messages to them are 'tagged' with completely random data.
* Users who do not care about anonymity can specify a very precise prefix for each of their addresses, allowing them to download only messages destined for them.
* Other users can achieve a balance between anonymity and convenience.

== Reasoning behind the proposal ==

Currently, Bitmessage clients such as PyBitmessage act as full nodes in the Bitmessage peer-to-peer network. This involves transmitting, receiving, and processing a large amount of data. This workload may be manageable for more powerful devices, such as laptops, desktops, and servers, but it is typically too great for less powerful devices such as mobile phones and tablets to reasonably handle. This is particularly true in cases where mobile internet connections mean that bandwidth is very expensive. Because of this, a high proportion of the computing devices in use today are not suitable to act as full nodes in the Bitmessage peer-to-peer network. Therefore, in order for the users of these devices to be able to use Bitmessage, there must be some way of doing so that does not involve their device acting as a full node in the network.

One approach to solving this problem is the creation of 'lite' clients. These are Bitmessage clients which do not do all the processing and transmission work that full nodes do. Instead, the burden of this work is shifted to servers, which supply lite clients with the data they need and relay data sent by lite clients to the rest of the Bitmessage network. Thankfully the design of Bitmessage allows this to be implemented in such a way that very little security is lost. All message encryption and decryption can be done locally by the lite client, and it can retain sole control over its addresses and cryptographic keys. Data provided by servers can be cryptographically authenticated as being correct, and data sent to servers can be disseminated to several at once to ensure that it reaches the rest of the network.

However, this arrangement does require one significant compromise. In order for 'lite' clients such as mobile phones to receive messages, there needs to be some way for them to identify messages that have been sent to their addresses. Full nodes in Bitmessage do not require this because they receive and automatically attempt to decrypt all messages in the streams (segments of the network) which they are a part of. As described above however, the work required for this approach is too great for many devices, and therefore there must be some way for lite clients to identify which messages are destined for them, so that they can request those messages from a server.

== Proposed changes ==

The pubkey of every hashed address contains two new fields: a 'prefix mask' field and a 'prefix' field. The value of the prefix mask field is determined by the owner of the pubkey. The prefix field is set randomly. The value of the prefix mask field defines what balance of efficiency and anonymity the owner of that address wants. If the owner of the pubkey wants maximum anonymity, then all the bits in the prefix mask field will be set to 0, allowing the prefix to be ignored entirely. On the other extreme, if the owner of the pubkey wants maximum efficiency in retrieving data at the expense of anonymity then all bits in the prefix mask field will be set to 1. Each additional bit set to one halves the proportion of the total data set that you will have to request in order to receive data destined for you. The positions of the bits set to 1 in the prefix mask field are selected randomly.

All msgs have a 'prefix nonce' as part of their unencrypted data. This prefix nonce is always of a fixed length (32 bits) to avoid leaking information. When sending a msg to an address, the sending client examines the destination pubkey's prefix mask field and records which bit positions (if any) are set to 1. The sending client will then set the bits in the corresponding positions of the prefix nonce to match those found in the destination pubkey's prefix field. The remaining bits of the prefix nonce are set randomly.

When a lite client wishes to retrieve msgs from the network, it makes a request to a server for all msgs which have a prefix nonce where the bits set to 1 in the prefix mask of the receiving address's pubkey match the prefix field of the receiving pubkey.

=== Pubkey object ===
Under this proposal, the encrypted part of a pubkey object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#pubkey.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
|-
| 4||[[Protocol_specification#Pubkey_bitfield_features|behavior bitfield]]||uint32_t||A bitfield of optional behaviors and features that can be expected from the node receiving the message.
|-
| 4||prefix_mask||uint32_t||A bit mask that defines which bits from the prefix field should be used.
|-
| 4||prefix||uint32_t||The prefix value. Contains 32 randomly set bits.
|-
| 64||public signing key||uchar[]||The ECC public key used for signing (uncompressed format; normally prepended with \x04 )
|-
| 64||public encryption key||uchar[]||The ECC public key used for encryption (uncompressed format; normally prepended with \x04 )
|-
| 1+||nonce_trials_per_byte||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is the average number of nonce trials a node will have to perform to meet the Proof of Work requirement. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||extra_bytes||var_int||Used to calculate the difficulty target of messages accepted by this node. The higher this value, the more difficult the Proof of Work must be before this individual will accept the message. This number is added to the data length to make sending small messages more difficult. 1000 is the network minimum so any lower values will be automatically raised to 1000.
|-
| 1+||sig_length||var_int||Length of the signature
|-
| sig_length||signature||uchar[]||The ECDSA signature which covers everything from the object header starting with the time, then appended with the decrypted data down to the extra_bytes. This was changed in protocol v3. 
|}

=== Msg object ===
Under this proposal, a msg object would be composed as follows. This can be compared to the current specification found at https://bitmessage.org/wiki/Protocol_specification#msg.

{|class="wikitable"
! Field Size !! Description !! Data type !! Comments
|-
| 4||prefixNonce||uint32_t||The prefix nonce.
|-
| ?||encrypted||uchar[]||Encrypted data. See [[Protocol_specification#Encrypted_payload|Encrypted payload]]. See also [[Protocol_specification#Unencrypted_Message_Data|Unencrypted Message Data Format]]
|}

== Examples ==

===Example 1===
If Alice does not care about anonymity, she can have her address's pubkey specify that the full 32 bits of the prefix field should be used. This is done by setting all bits of the pubkey's prefix mask field to 1. This means that all msgs sent to her will effectively be uniquely tagged for her address (there are roughly 4 billion possible prefix values). This will allow Alice to only download msgs that are destined for her, giving her maximum usability and convenience at the expense of anonymity.

===Example 2===
If Bob wants to achieve a balance between anonymity and convenience, he can have his address's pubkey specify that 4 bits of the prefix field should be used. This is done by setting 4 randomly selected bits of the pubkey's prefix mask field to 1. This means that all msgs sent to his address will have a prefix nonce where the 4 bits in the positions set to 1 in the prefix mask will match the corresponding bits in his pubkey's prefix field. A random distribution of prefixes in a stream should mean that roughly 6.25% of msgs in a stream will have a prefix nonce where the bits in those positions match the values of those bits in his pubkey's prefix field. Therefore Bob will be able to download roughly 6.25% of the msgs in a stream and still be guaranteed to receive all msgs destined for him.

===Example 3===
If Carol wants maximum anonymity then she can have her address's pubkey specify that zero bits of the prefix field should be used. This is done by setting all of the bits of the pubkey's prefix mask field to 0. This means that msgs sent to her will contain no identifying information, only a totally random prefix nonce. Carol will have to download and attempt to decrypt all msgs in a stream in order to be sure of receiving all msgs destined for her.

== Notes ==

===Other object types===
Prefix filtering is not required for broadcasts, pubkeys, or getpubkeys. These object types are already tagged with an identifier.

===Lite client pubkey retrieval===
How can lite clients retrieve pubkeys without telling the servers who they are talking to? 
In the short to medium term (until streams are implemented), it will be perfectly viable for lite clients to simply download all pubkeys in stream 1. A pubkey is 392 bytes in size. If we assume an average of 5 addresses per user, we could support a 10x increase in the number of active users (from roughly 1,000 to 10,000), and lite clients would still only have to download 19MB worth of pubkey data. This is only a rough approximation, but it seems reasonably clear that the amount of data to download and store would be acceptable.

===Lite client pubkey dissemination===
How can lite clients respond to getpubkey requests? 
Lite clients can do this in two ways: 
1. They can make periodic requests to full node servers to check for any getpubkeys for any of their addresses and then respond to any getpubkeys they receive by re-disseminating the relevant pubkeys. 
2. They can periodically re-disseminate the pubkeys for each of their addresses, with the re-dissemiantion period determined by the time to live of the pubkeys they create. This will ensure that the pubkeys for all the lite client's addresses are always available in the stream(s) they are a part of.

===Non-hashed addresses===
If non-hashed addresses are introduced in the future, this scheme will not apply to them, as they will not have pubkeys. Users of non-hashed addresses will still need to download and attempt to decrypt all msgs in each stream in which they have an address. This makes sense though, as the whole point of introducing non-hashed addresses is to give people the ability to choose greater anonymity at the expense of convenience.

===Streams===
The system described in this proposal is intended to be complementary to streams (i.e. both should be implemented). It is not intended to replace them.