diff --git a/docs/How-To-Guides/add_ibft_validator.md b/docs/How-To-Guides/add_ibft_validator.md new file mode 100644 index 000000000..8c16336c0 --- /dev/null +++ b/docs/How-To-Guides/add_ibft_validator.md @@ -0,0 +1,291 @@ +# Adding and removing IBFT validators + +Over the lifetime of an IBFT network, validators will need to be added and removed as authorities change. +Here we will showcase adding a new validator to an IBFT network, as well as removing an existing one. + +## Adding a node to the validator set + +Adding a node to the IBFT validator set is relatively easy once a node is part of the network. +It does not matter whether the node is already online or not, as the process to add the new node as a validator only +needs the *existing* validators. + +!!! warning + If you are adding multiple validators before they are brought online, make sure you don't go over the BFT limit and cause the chain to stop progressing. + +Adding a new validator requires that a majority of existing validators propose the new node to be added. This is +achieved by calling the `propose` RPC method with the value `true` and replacing the address to your required one: + + ```bash + $ geth attach /qdata/dd/geth.ipc + + > istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true); + null + ``` + +This indicates that the current node wishes to add address `0xb131288f355bc27090e542ae0be213c20350b767` as a new +validator. + +### Example + +You can find the resources required to run the examples in the +[quorum-examples](https://github.com/jpmorganchase/quorum-examples/tree/master/examples/ibft_validator_set_changes) +repository. + +1. The examples use `docker-compose` for the container definitions. If you are following along by copying the commands + described, then it is important to set the project name for Docker Compose, or to remember to change the prefix for + your directory. See [Docker documentation](https://docs.docker.com/compose/reference/envvars/#compose_project_name) + for more details. + + To set the project name, run the following: + ```bash + $ export COMPOSE_PROJECT_NAME=addnode + ``` + +2. Bring up the network, which contains 7 nodes, of which 6 are validators. + + ```bash + $ docker-compose -f ibft-6-validators.yml up + ``` + + We will be adding the 7th node as a validator. You may notice in the logs of node 7 messages along the lines of + `node7_1 | WARN [01-20|10:37:16.034] Block sealing failed err=unauthorized`. This is because + the node was started up with minting enabled, but doesn't have the authority to create blocks, and so throws this + error. + +3. Now we need to propose node 7 as a new proposer from the existing nodes. + + !!! note + Remember, you could do this stage before starting node 7 in your network + + We need a majority of existing validators to propose the new node before the changes will take effect. + + Lets start with node 1 and see what happens: + + ```bash + # Propose node 7 from node 1 + $ docker exec -it addnode_node1_1 geth --exec 'istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);' attach /qdata/dd/geth.ipc + null + + # Wait about 5 seconds, and then run: + $ docker exec -it addnode_node1_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc + { + epoch: 30000, + hash: "0xf814863d809ce3a683ee0a2197b15a8152d2696fc9c4e47cd82d0bd5cdaa3e45", + number: 269, + policy: 0, + tally: { + 0xb131288f355bc27090e542ae0be213c20350b767: { + authorize: true, + votes: 1 + } + }, + validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"], + votes: [{ + address: "0xb131288f355bc27090e542ae0be213c20350b767", + authorize: true, + block: 268, + validator: "0xd8dba507e85f116b1f7e231ca8525fc9008a6966" + }] + } + ``` + + Let's break this down. + Firstly, we proposed the address `0xb131288f355bc27090e542ae0be213c20350b767` to be added; that is what the `true` + parameter is for. If we had set it to `false`, that means we want to remove an existing validator with that address. + + Secondly, we fetched the current snapshot, which gives us an insight into the current running state of the voting. + We can see that the new address has 1 vote under the `tally` section, and that one vote is described under the + `votes` section. So we know our vote was registered! + +4. Let's run this from node 2 and see similar results: + + ```bash + $ docker exec -it addnode_node2_1 geth --exec 'istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);' attach /qdata/dd/geth.ipc + null + + # Again, you may have to wait 5 - 10 seconds for the snapshot to show the vote + $ docker exec -it addnode_node2_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc + { + epoch: 30000, + hash: "0x93efcd458f3b875902a4532bb77d5e7ebb701791ea95486ecd58baf682312d74", + number: 391, + policy: 0, + tally: { + 0xb131288f355bc27090e542ae0be213c20350b767: { + authorize: true, + votes: 2 + } + }, + validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"], + votes: [{ + address: "0xb131288f355bc27090e542ae0be213c20350b767", + authorize: true, + block: 388, + validator: "0xd8dba507e85f116b1f7e231ca8525fc9008a6966" + }, { + address: "0xb131288f355bc27090e542ae0be213c20350b767", + authorize: true, + block: 390, + validator: "0x6571d97f340c8495b661a823f2c2145ca47d63c2" + }] + } + ``` + + True to form, we have the second vote registered! + +5. Ok, let's finally vote on nodes 3 and 4. + + ```bash + $ docker exec -it addnode_node3_1 geth --exec 'istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);' attach /qdata/dd/geth.ipc + null + + $ docker exec -it addnode_node4_1 geth --exec 'istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);' attach /qdata/dd/geth.ipc + null + ``` + +6. Now we have a majority of votes, let's check the snapshot again: + + ```bash + docker exec -it addnode_node1_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc + { + epoch: 30000, + hash: "0xd4234184538297f71f5b7024a2e11f51f06b4f569ebd9e3644abd391b8c66101", + number: 656, + policy: 0, + tally: {}, + validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", "0xb131288f355bc27090e542ae0be213c20350b767", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"], + votes: [] + } + ``` + + We can see that the votes have now been wiped clean, ready for a new round. Additionally, the address we were adding, + `0xb131288f355bc27090e542ae0be213c20350b767` now exists within the `validators` list! + Lastly, the `unauthorized` messages that node 7 was giving before has stopped, as it now has the authority to mint + blocks. + +## Removing a node from the validator set + +Removing a validator is very similar to adding a node, but this time we want to propose nodes with the value `false`, +to indicate we are deauthorising them. It does not matter whether the node is still online or not, as it doesn't +require any input from the node being removed. + +!!! warning + Be aware when removing nodes that cross the BFT boundary, e.g. going from 10 validators to 9, as this may impact the chains ability to progress if other nodes are offline + +Removing a new validator requires that a majority of existing validators propose the new node to be removed. This is +achieved by calling the `propose` RPC method with the value `false` and replacing the address to your required one: + + ```bash + $ geth attach /qdata/dd/geth.ipc + + > istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", false); + null + ``` + +### Example + +You can find the resources required to run the examples in the +[quorum-examples](https://github.com/jpmorganchase/quorum-examples/tree/master/examples/ibft_validator_set_changes) +repository. + +1. The examples use `docker-compose` for the container definitions. If you are following along by copying the commands + described, then it is important to set the project name for Docker Compose, or to remember to change the prefix for + your directory. See [Docker documentation](https://docs.docker.com/compose/reference/envvars/#compose_project_name) + for more details. + + To set the project name, run the following: + ```bash + $ export COMPOSE_PROJECT_NAME=addnode + ``` + +2. Bring up the network, which contains 7 nodes, of which 6 are validators. + + ```bash + # Set the environment variable for docker-compose + $ export COMPOSE_PROJECT_NAME=addnode + + # Start the 7 node network, of which 6 are validators + $ docker-compose -f ibft-6-validators.yml up + ``` + +3. Now we need to propose node 6 as the node to remove. + + !!! note + We need a majority of existing validators to propose the new node before the changes will take effect. + + Lets start with node 1 and see what happens: + + ```bash + # Propose node 7 from node 1 + $ docker exec -it addnode_node1_1 geth --exec 'istanbul.propose("0x8157d4437104e3b8df4451a85f7b2438ef6699ff", false);' attach /qdata/dd/geth.ipc + null + + # Wait about 5 seconds, and then run: + $ docker exec -it addnode_node1_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc + { + epoch: 30000, + hash: "0xba9f9b72cad90ae8aee39f352b45f21d5ed5535b4479743e3f39b231fd717792", + number: 140, + policy: 0, + tally: { + 0x8157d4437104e3b8df4451a85f7b2438ef6699ff: { + authorize: false, + votes: 1 + } + }, + validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"], + votes: [{ + address: "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", + authorize: false, + block: 136, + validator: "0xd8dba507e85f116b1f7e231ca8525fc9008a6966" + }] + } + ``` + + Let's break this down. + Firstly, we proposed the address `0x8157d4437104e3b8df4451a85f7b2438ef6699ff` to be removed; that is what the + `false` parameter is for. + + Secondly, we fetched the current snapshot, which gives us an insight into the current running state of the voting. + We can see that the proposed address has 1 vote under the `tally` section, and that one vote is described under the + `votes` section. Here, the `authorize` section is set to `false`, which is inline with our proposal to *remove* the + validator. + +4. We need to get a majority, so let's run the proposal on 3 more nodes: + + ```bash + $ docker exec -it addnode_node2_1 geth --exec 'istanbul.propose("0x8157d4437104e3b8df4451a85f7b2438ef6699ff", false);' attach /qdata/dd/geth.ipc + null + + $ docker exec -it addnode_node3_1 geth --exec 'istanbul.propose("0x8157d4437104e3b8df4451a85f7b2438ef6699ff", false);' attach /qdata/dd/geth.ipc + null + + $ docker exec -it addnode_node4_1 geth --exec 'istanbul.propose("0x8157d4437104e3b8df4451a85f7b2438ef6699ff", false);' attach /qdata/dd/geth.ipc + null + ``` + +5. Let's check the snapshot now all the required votes are in: + + ```bash + $ docker exec -it addnode_node1_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc + { + epoch: 30000, + hash: "0x25815a32b086926875ea2c44686e4b20effabc731b2b121ebf0e0f395101eea5", + number: 470, + policy: 0, + tally: {}, + validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"], + votes: [] + } + ``` + + The validator has been removed from the `validators` list, and we are left with the other 5 still present. You will + also see in the logs of node 6 a message like + `node6_1 | WARN [01-20|11:35:52.044] Block sealing failed err=unauthorized`. This is because it is still minting + blocks, but realises it does not have the authority to push them to any of the other nodes on the network (you will + also see this message for node 7, which was never authorised but still set up to mine). + +## See also + +- [Adding a new node to the network](/How-To-Guides/adding_nodes) \ No newline at end of file diff --git a/docs/How-To-Guides/add_node_examples.md b/docs/How-To-Guides/add_node_examples.md new file mode 100644 index 000000000..047dc0366 --- /dev/null +++ b/docs/How-To-Guides/add_node_examples.md @@ -0,0 +1,388 @@ +# Node addition examples + +Below are some scenarios for adding a new node into a network, with a mix of different options such as +consensus algorithm, permissioning and discovery. + +You can find the resources required to run the examples in the +[quorum-examples](https://github.com/jpmorganchase/quorum-examples/tree/master/examples/adding_nodes) repository. +Checkout the repository through `git` or otherwise download all the resources your local machine to follow along. + +The examples use `docker-compose` for the container definitions. If you are following along by copying the commands +described, then it is important to set the project name for Docker Compose, or to remember to change the prefix for +your directory. See [Docker documentation](https://docs.docker.com/compose/reference/envvars/#compose_project_name) +for more details. + +To set the project name, run the following: +```bash +$ export COMPOSE_PROJECT_NAME=addnode +``` + +## Non-permimssioned IBFT with discovery + +An example using IBFT, no permissioning and discover enabled via a bootnode. +There are no static peers in this network; instead, every node is set to talk to node 1 via the CLI flag +`--bootnodes enode://ac6b1096ca56b9f6d004b779ae3728bf83f8e22453404cc3cef16a3d9b96608bc67c4b30db88e0a5a6c6390213f7acbe1153ff6d23ce57380104288ae19373ef@172.16.239.11:21000`. +Node 1 will forward the details of all the nodes it knows about (in this case, everyone) and they will then initiate their +own connections. + +1. Bring up an initial network of 6 nodes. + + ```bash + # Ensure any old network is removed + $ docker-compose -f ibft-non-perm-bootnode.yml down + + # Bring up 6 nodes + $ docker-compose -f ibft-non-perm-bootnode.yml up node1 node2 node3 node4 node5 node6 + ``` + +2. Send in a public transaction and check it is minted. + + !!! note + * The block creation period is set to 2 seconds, so you may have to wait upto that amount of time for the transaction to be minted. + * The transaction hashes will likely be different, but the contract addresses will be the same for your network. + + ```bash + # Send in the transaction + $ docker exec -it addnode_node1_1 geth --exec 'loadScript("/examples/public-contract.js")' attach /qdata/dd/geth.ipc + Contract transaction send: TransactionHash: 0xd1bf0c15546802e5a121f79d0d8e6f0fa45d4961ef8ab9598885d28084cfa909 waiting to be mined... + true + + # Retrieve the value of the contract + $ docker exec -it addnode_node1_1 geth --exec 'var private = eth.contract([{"constant":true,"inputs":[],"name":"get","outputs":[{"name":"retVal","type":"uint256"}],"payable":false,"type":"function"}]).at("0x1932c48b2bf8102ba33b4a6b545c32236e342f34"); private.get();' attach /qdata/dd/geth.ipc + 42 + ``` + + We created a transaction, in this case with hash `0xd1bf0c15546802e5a121f79d0d8e6f0fa45d4961ef8ab9598885d28084cfa909`, + and then retrieved its value, which was set to be `42`. + +3. Bring up the last node. This node also has its bootnodes set to be node 1, so at startup will try to establish a +connection to node 1 only. After this, node 1 will share which nodes it knows about, and node 7 can then initiate +connections with those peers. + + ```bash + # Bring up node 7 + $ docker-compose -f ibft-non-perm-bootnode.yml up node7 + ``` + +4. Let's check to see if the nodes are in sync. If they are, they will have similar block numbers, which is enough for +this example; there are other ways to tell if nodes are on the same chain, e.g. matching block hashes. + + !!! note + Depending on timing, the second may have an extra block or two. + + ```bash + # Fetch the latest block number for node 1 + $ docker exec -it addnode_node1_1 geth --exec 'eth.blockNumber' attach /qdata/dd/geth.ipc + 45 + + # Fetch the latest block number for node 7 + $ docker exec -it addnode_node7_1 geth --exec 'eth.blockNumber' attach /qdata/dd/geth.ipc + 45 + ``` + +5. We can check that the transaction and contract we sent earlier now exist on node 7. + + ```bash + $ docker exec -it addnode_node7_1 geth --exec 'var private = eth.contract([{"constant":true,"inputs":[],"name":"get","outputs":[{"name":"retVal","type":"uint256"}],"payable":false,"type":"function"}]).at("0x1932c48b2bf8102ba33b4a6b545c32236e342f34"); private.get();' attach /qdata/dd/geth.ipc + 42 + ``` + +6. To be sure we have two way communication, let's send a transaction from node 7 to the network. + + ```bash + $ docker exec -it addnode_node7_1 geth --exec 'loadScript("/examples/public-contract.js")' attach /qdata/dd/geth.ipc + Contract transaction send: TransactionHash: 0x84cefc3aab8ce5797dc73c70db604e5c8830fc7c2cf215876eb34fff533e2725 waiting to be mined... + true + ``` + +7. Finally, we can check if the transaction was minted and the contract executed on each node. + + ```bash + # Check on node 1 + $ docker exec -it addnode_node1_1 geth --exec 'var private = eth.contract([{"constant":true,"inputs":[],"name":"get","outputs":[{"name":"retVal","type":"uint256"}],"payable":false,"type":"function"}]).at("0x1349f3e1b8d71effb47b840594ff27da7e603d17"); private.get();' attach /qdata/dd/geth.ipc + 42 + + # Check on node 7 + $ docker exec -it addnode_node7_1 geth --exec 'var private = eth.contract([{"constant":true,"inputs":[],"name":"get","outputs":[{"name":"retVal","type":"uint256"}],"payable":false,"type":"function"}]).at("0x1349f3e1b8d71effb47b840594ff27da7e603d17"); private.get();' attach /qdata/dd/geth.ipc + 42 + ``` + +And that's it. We deployed a working 6 node network, and then added a 7th node afterwards; this 7th node was able to +read existing public data, as well as deploy its own transactions and contracts for others to see! + +## Non-permissioned RAFT with discovery disabled + +This example walks through adding a new node to a RAFT network. This network does not have permissioning for the +Ethereum peer-to-peer layer, and makes it connections solely based on who is listed in the nodes `static-nodes.json` +file. + +1. Bring up an initial network of 6 nodes. + + ```bash + # Ensure any old network is removed + $ docker-compose -f raft-non-perm-nodiscover.yml down + + # Bring up 6 nodes + $ docker-compose -f raft-non-perm-nodiscover.yml up node1 node2 node3 node4 node5 node6 + ``` + +2. Send in a public transaction and check it is minted. + + !!! note + * The transaction hashes will likely be different, but the contract addresses will be the same for your network. + + ```bash + # Send in the transaction + $ docker exec -it addnode_node1_1 geth --exec 'loadScript("/examples/public-contract.js")' attach /qdata/dd/geth.ipc + Contract transaction send: TransactionHash: 0xd1bf0c15546802e5a121f79d0d8e6f0fa45d4961ef8ab9598885d28084cfa909 waiting to be mined... + true + + # Retrieve the value of the contract + $ docker exec -it addnode_node1_1 geth --exec 'var private = eth.contract([{"constant":true,"inputs":[],"name":"get","outputs":[{"name":"retVal","type":"uint256"}],"payable":false,"type":"function"}]).at("0x1932c48b2bf8102ba33b4a6b545c32236e342f34"); private.get();' attach /qdata/dd/geth.ipc + 42 + ``` + + We created a transaction, in this case with hash `0xd1bf0c15546802e5a121f79d0d8e6f0fa45d4961ef8ab9598885d28084cfa909`, + and then retrieved its value, which was set to be `42`. + +3. We need to add the new peer to the RAFT network before it joins, otherwise the existing nodes will reject it from +the RAFT communication layer; we also need to know what ID the new node should join with. + + ```bash + # Add the new node + $ docker exec -it addnode_node1_1 geth --exec 'raft.addPeer("enode://239c1f044a2b03b6c4713109af036b775c5418fe4ca63b04b1ce00124af00ddab7cc088fc46020cdc783b6207efe624551be4c06a994993d8d70f684688fb7cf@172.16.239.17:21000?discport=0&raftport=50400")' attach /qdata/dd/geth.ipc + 7 + ``` + + The return value is the RAFT ID of the new node. When the node joins the network for the first time, it will need + this ID number handy. If it was lost, you can always view the full network, including IDs, by running the + `raft.cluster` command on an existing node. + +4. Bring up the last node. Here, we pass the newly created ID number as a flag into the startup of node 7. This lets +the node know to not bootstrap a new network from the contents of `static-nodes.json`, but to connect to an existing +node there are fetch any bootstrap information. + + ```bash + # Bring up node 7 + $ QUORUM_GETH_ARGS="--raftjoinexisting 7" docker-compose -f raft-non-perm-nodiscover.yml up node7 + ``` + +5. Let's check to see if the nodes are in sync. We can do by seeing if we have the contract that we viewer earlier on +node 7. + + ```bash + # Fetch the contracts value on node 7 + $ docker exec -it addnode_node7_1 geth --exec 'var private = eth.contract([{"constant":true,"inputs":[],"name":"get","outputs":[{"name":"retVal","type":"uint256"}],"payable":false,"type":"function"}]).at("0x1932c48b2bf8102ba33b4a6b545c32236e342f34"); private.get();' attach /qdata/dd/geth.ipc + 42 + ``` + +6. To be sure we have two way communication, let's send a transaction from node 7 to the network. + + ```bash + $ docker exec -it addnode_node7_1 geth --exec 'loadScript("/examples/public-contract.js")' attach /qdata/dd/geth.ipc + Contract transaction send: TransactionHash: 0x84cefc3aab8ce5797dc73c70db604e5c8830fc7c2cf215876eb34fff533e2725 waiting to be mined... + true + ``` + +7. Finally, we can check if the transaction was minted and the contract executed on each node. + + ```bash + # Check on node 1 + $ docker exec -it addnode_node1_1 geth --exec 'var private = eth.contract([{"constant":true,"inputs":[],"name":"get","outputs":[{"name":"retVal","type":"uint256"}],"payable":false,"type":"function"}]).at("0x1349f3e1b8d71effb47b840594ff27da7e603d17"); private.get();' attach /qdata/dd/geth.ipc + 42 + + # Check on node 7 + $ docker exec -it addnode_node7_1 geth --exec 'var private = eth.contract([{"constant":true,"inputs":[],"name":"get","outputs":[{"name":"retVal","type":"uint256"}],"payable":false,"type":"function"}]).at("0x1349f3e1b8d71effb47b840594ff27da7e603d17"); private.get();' attach /qdata/dd/geth.ipc + 42 + ``` + +And that's it. We deployed a working 6 node network, and then added a 7th node afterwards; this 7th node was able to +read existing public data, as well as deploy its own transactions and contracts for others to see! + +## Permissioned RAFT with discovery disabled + +This example walks through adding a new node to a RAFT network. This network does have permissioning enabled for the +Ethereum peer-to-peer layer; this means that for any Ethereum tasks, such as syncing the initial blockchain or +propagating transactions, the node must appear is others nodes' `permissioned-nodes.json` file. + +1. Bring up an initial network of 6 nodes. + + ```bash + # Ensure any old network is removed + $ docker-compose -f raft-perm-nodiscover.yml down + + # Bring up 6 nodes + $ docker-compose -f raft-perm-nodiscover.yml up node1 node2 node3 node4 node5 node6 + ``` + +2. Send in a public transaction and check it is minted. + + !!! note + * The transaction hashes will likely be different, but the contract addresses will be the same for your network. + + ```bash + # Send in the transaction + $ docker exec -it addnode_node1_1 geth --exec 'loadScript("/examples/public-contract.js")' attach /qdata/dd/geth.ipc + Contract transaction send: TransactionHash: 0xd1bf0c15546802e5a121f79d0d8e6f0fa45d4961ef8ab9598885d28084cfa909 waiting to be mined... + true + + # Retrieve the value of the contract + $ docker exec -it addnode_node1_1 geth --exec 'var private = eth.contract([{"constant":true,"inputs":[],"name":"get","outputs":[{"name":"retVal","type":"uint256"}],"payable":false,"type":"function"}]).at("0x1932c48b2bf8102ba33b4a6b545c32236e342f34"); private.get();' attach /qdata/dd/geth.ipc + 42 + ``` + + We created a transaction, in this case with hash `0xd1bf0c15546802e5a121f79d0d8e6f0fa45d4961ef8ab9598885d28084cfa909`, + and then retrieved its value, which was set to be `42`. + +3. We need to add the new peer to the RAFT network before it joins, otherwise the existing nodes will reject it from +the RAFT communication layer; we also need to know what ID the new node should join with. + + ```bash + # Add the new node + $ docker exec -it addnode_node1_1 geth --exec 'raft.addPeer("enode://239c1f044a2b03b6c4713109af036b775c5418fe4ca63b04b1ce00124af00ddab7cc088fc46020cdc783b6207efe624551be4c06a994993d8d70f684688fb7cf@172.16.239.17:21000?discport=0&raftport=50400")' attach /qdata/dd/geth.ipc + 7 + ``` + + The return value is the RAFT ID of the new node. When the node joins the network for the first time, it will need + this ID number handy. If it was lost, you can always view the full network, including IDs, by running the + `raft.cluster` command on an existing node. + +4. Bring up the last node. Here, we pass the newly created ID number as a flag into the startup of node 7. This lets +the node know to not bootstrap a new network from the contents of `static-nodes.json`, but to connect to an existing +node there are fetch any bootstrap information. + + ```bash + # Bring up node 7 + $ QUORUM_GETH_ARGS="--raftjoinexisting 7" docker-compose -f raft-non-perm-nodiscover.yml up node7 + ``` + +5. Let's check to see if the nodes are in sync. We can do by seeing if we have the contract that we viewer earlier on +node 7. + + ```bash + # Fetch the contracts value on node 7 + $ docker exec -it addnode_node7_1 geth --exec 'var private = eth.contract([{"constant":true,"inputs":[],"name":"get","outputs":[{"name":"retVal","type":"uint256"}],"payable":false,"type":"function"}]).at("0x1932c48b2bf8102ba33b4a6b545c32236e342f34"); private.get();' attach /qdata/dd/geth.ipc + 0 + ``` + + The value here is `0`, not the expected `42`! Node 7 is unable to sync the blockchain because the other peers in the + network are refusing to allow connections from node 7, due to it being missing in the `permissioned-nodes.json` file. + + This does not affect the RAFT layer, so if node 7 was already is sync, it could still receive new blocks; this is + okay though, since it would be permissioned on the RAFT side by virtue of being part of the RAFT cluster. + +6. Let's update the permissioned nodes list on node 1, which will allow node 7 to connect to it. + + ```bash + $ docker exec -it addnode_node1_1 cp /extradata/static-nodes-7.json /qdata/dd/permissioned-nodes.json + $ + ``` + +7. Node 7 should now be synced up through node 1. Let's see if we can see the contract we made earlier. + + !!! note + Quorum attempts to re-establish nodes every 30 seconds, so you may have to wait for the sync to happen. + + ```bash + $ docker exec -it addnode_node7_1 geth --exec 'var private = eth.contract([{"constant":true,"inputs":[],"name":"get","outputs":[{"name":"retVal","type":"uint256"}],"payable":false,"type":"function"}]).at("0x1932c48b2bf8102ba33b4a6b545c32236e342f34"); private.get();' attach /qdata/dd/geth.ipc + 42 + ``` + +8. To be sure we have two way communication, let's send a transaction from node 7 to the network. + + ```bash + $ docker exec -it addnode_node7_1 geth --exec 'loadScript("/examples/public-contract.js")' attach /qdata/dd/geth.ipc + Contract transaction send: TransactionHash: 0x84cefc3aab8ce5797dc73c70db604e5c8830fc7c2cf215876eb34fff533e2725 waiting to be mined... + true + ``` + +9. Finally, we can check if the transaction was minted and the contract executed on each node. + + ```bash + # Check on node 1 + $ docker exec -it addnode_node1_1 geth --exec 'var private = eth.contract([{"constant":true,"inputs":[],"name":"get","outputs":[{"name":"retVal","type":"uint256"}],"payable":false,"type":"function"}]).at("0x1349f3e1b8d71effb47b840594ff27da7e603d17"); private.get();' attach /qdata/dd/geth.ipc + 42 + + # Check on node 7 + $ docker exec -it addnode_node7_1 geth --exec 'var private = eth.contract([{"constant":true,"inputs":[],"name":"get","outputs":[{"name":"retVal","type":"uint256"}],"payable":false,"type":"function"}]).at("0x1349f3e1b8d71effb47b840594ff27da7e603d17"); private.get();' attach /qdata/dd/geth.ipc + 42 + ``` + +And that's it. We deployed a working 6 node network, and then added a 7th node afterwards; this 7th node was able to + read existing public data, as well as deploy its own transactions and contracts for others to see! + +## Adding a Private Transaction Manager + +This is a simple example of adding a new Tessera instance to an existing network. For simplicity, +the steps to add the Quorum node are omitted, but are those followed in the IBFT example. +Here, a Tessera node is added without any of the discovery options specified, meaning that the +IP Whitelist isn't used, nor is key discovery disabled. + +1. Start up the initial 6 node network. + + ```bash + # Ensure any old network is removed + $ docker-compose -f tessera-add.yml down + + # Bring up 6 nodes + $ docker-compose -f tessera-add.yml up node1 node2 node3 node4 node5 node6 + ``` + +2. We can verify that private transactions can be sent by sending one from node 1 to node 6. +We can also see that since node 7 doesn't exist yet, we can't send private transactions to it. + + ```bash + # Send a private transaction from node 1 to node 6 + $ docker exec -it addnode_node1_1 geth --exec 'loadScript("/examples/private-contract-6.js")' attach /qdata/dd/geth.ipc + Contract transaction send: TransactionHash: 0xc8a5de4bb79d4a8c3c1156917968ca9b2965f2514732fc1cff357ec999b9aba4 waiting to be mined... + true + # Success! + + $ docker exec -it addnode_node1_1 geth --exec 'loadScript("/examples/private-contract-7.js")' attach /qdata/dd/geth.ipc + err creating contract Error: Non-200 status code: &{Status:404 Not Found StatusCode:404 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Server:[Jetty(9.4.z-SNAPSHOT)] Date:[Thu, 16 Jan 2020 12:44:19 GMT] Content-Type:[text/plain] Content-Length:[73]] Body:0xc028e87d40 ContentLength:73 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc000287200 TLS:} + true + # An expected failure. The script content didn't succeed, but the script itself was run okay, so true was still returned + ``` + +3. Let's first bring up node 7, then we can inspect what is happening and the configuration used. + + ```bash + # Bring up node 7 + $ docker-compose -f tessera-add.yml up node7 + + $ docker exec -it addnode_node7_1 cat /qdata/tm/tessera-config.json + # ...some output... + ``` + + The last command will output Tessera 7's configuration. + The pieces we are interested in here are the following: + + ```json + { + "useWhiteList": false, + "peer": [ + { + "url": "http://txmanager1:9000" + } + ], + ... + } + ``` + + We can see that the whitelist is not enabled, discovery is not specified so defaults to enabled, + and we have a single peer to start off with, which is node 1. + This is all that is needed to connect to an existing network. Shortly after starting up, Tessera + will ask node 1 about all it's peers, and then will keep a record of them for it's own use. From + then on, all the nodes will know about node 7 and can send private transactions to it. + +4. Let's try it! Let's send a private transaction from node 1 to the newly added node 7. + + ```bash + # Sending a transaction from node 1 to node 7 + $ docker exec -it addnode_node1_1 geth --exec 'loadScript("/examples/private-contract-7.js")' attach /qdata/dd/geth.ipc + Contract transaction send: TransactionHash: 0x3e3b50768ffdb51979677ddb58f48abdabb82a3fd4f0bac5b3d1ad8014e954e9 waiting to be mined... + true + ``` + + We got a success this time! Tessera 7 has been accepted into the network and can interact with the + other existing nodes. \ No newline at end of file diff --git a/docs/How-To-Guides/adding_nodes.md b/docs/How-To-Guides/adding_nodes.md new file mode 100644 index 000000000..3201c922b --- /dev/null +++ b/docs/How-To-Guides/adding_nodes.md @@ -0,0 +1,227 @@ +# Adding nodes to the network + +Adding new nodes to an existing network can range from a common occurence to never happening. +In public blockchains, such as the Ethereum Mainnet, new nodes continuously join and talk to the existing network. +In permissioned blockchains, this may not happen as often, but it still an important task to achieve as your network +evolves. + +When adding new nodes to the network, it is important understand that the Quorum network and Private Transaction +Manager network are distinct and do not overlap in any way. Therefore, options applicable to one are not applicable to +the other. In some cases, they may have their own options to achieve similar tasks, but must be specified separately. + +## Prerequisites + +- [Quorum installed](/Getting%20Started/Installing.md) +- [Tessera/Constellation installed](/Getting%20Started/Installing.md) if using private transactions +- A running network (see [Creating a Network From Scratch](/Getting%20Started/Creating-A-Network-From-Scratch)) + +## Adding Quorum nodes + +Adding a new Quorum node is the most common operation, as you can choose to run a Quorum node with or without a Private +Transaction Manager, but rarely will one do the opposite. + +### Raft + +1. On an *existing* node, add the new peer to the raft network + ``` + > raft.addPeer("enode://239c1f044a2b03b6c4713109af036b775c5418fe4ca63b04b1ce00124af00ddab7cc088fc46020cdc783b6207efe624551be4c06a994993d8d70f684688fb7cf@127.0.0.1:21006?discport=0&raftport=50407") + 7 + ``` + + So in this example, our new node has a Raft ID of `7`. + +2. If you are using permissioning, or discovery for Ethereum p2p, please refer [here](#extra-options). + +3. We now need to initialise the new node with the network's genesis configuration. + + !!! note + Where you obtain this from will be dependent on the network. You may get it from an existing peer, or a network operator, or elsewhere entirely. + + Initialising the new node is exactly the same an the original nodes. + ```bash + $ geth --datadir qdata/dd7 init genesis.json + ``` + +4. Now we can start up the new node and let it sync with the network. The main difference now is the use of the +`--raftjoinexisting` flag, which lets the node know that it is joining an existing network, which is handled +differently internally. The Raft ID obtained in step 1 is passed as a parameter to this flag. + + ```bash + $ PRIVATE_CONFIG=ignore geth --datadir qdata/dd7 ... OTHER ARGS ... --raft --raftport 50407 --rpcport 22006 --port 21006 --raftjoinexisting 7 + ``` + + The new node is now up and running, and will start syncing the blockchain from existing peers. Once this has + completed, it can send new transactions just as any other peer. + +### IBFT/Clique + +Adding nodes to an IBFT/Clique network is a bit simpler, as it only needs to configure itself rather then be +pre-allocated on the network (permissioning aside). + +1. Initialise the new node with the network's genesis configuration. + + !!! note + Where you obtain this from will be dependent on the network. You may get it from an existing peer, or a network operator, or elsewhere entirely. + + Initialising the new node is exactly the same an the original nodes. + ```bash + $ geth --datadir qdata/dd7 init genesis.json + ``` + +2. If you are using permissioning or discovery for Ethereum peer-to-peer, please refer [here](#extra-options). + +3. Start the new node, pointing either to a `bootnode` or listing an existing peer in the `static-nodes.json` file. +Once a connection is established, the node will start syncing the blockchain, after which transactions can be sent. + +### Extra options + +Some options take effect regardless of the consensus mechanism used. + +#### Permissioned nodes + +If using the `permissioned-nodes.json` file for permissioning, then you must make sure this file is updated on all +nodes before the new node is able to communicate with existing nodes. You do not need to restart any nodes in +order for the changes to take effect. + +#### Static node connections + +If not using peer-to-peer node discovery (i.e. you have specified `--nodiscover`), then the only connections a node +made will be to peers defined in the `static-nodes.json` file. When adding a new node, you should make sure you have +peers defined in its `static-nodes.json` file. The more peers you have defined here, the better network connectivity +and fault tolerance you have. + +!!! note + * You do not need to update the existing peers static nodes for the connection to be established, although it is good practise to do so. + * You do not need to specify every peer in your static nodes file if you do not wish to connect to every peer directly. + +#### Peer-to-peer discovery + +If you are using discovery, then more options *in addition* to static nodes become available. + +- Any nodes that are connected to your peers, which at the start will be ones defined in the static node list, will +then be visible by you, allowing you to connect to them; this is done automatically. + +- You may specify any number of bootnodes, defined by the `--bootnodes` parameter. This takes a commas separated list +of enode URIs, similar to the `static-nodes.json` file. These act in the same way as static nodes, letting you connect +to them and then find out about other peers, whom you then connect to. + +!!! note + If you have discovery disabled, this means you will not try to find other nodes to connect to, but others can still find and connect to you. + +## Adding Private Transaction Managers + +In this tutorial, there will be no focus on the advanced features of adding a new Private Transaction Manager (PTM). +This tutorial uses [Tessera](https://github.com/jpmorganchase/tessera) for any examples. + +Adding a new node to the PTM is relatively straight forward, but there are a lot of extra options that can be used, +which is what will be explained here. + +### Adding a new PTM node + +In a basic setting, adding a new PTM node is as simple as making sure you have one of the existing nodes listed in your +peer list. + +In Tessera, this would equate to the following in the configuration file: +```json +{ + "peers": [ + { + "url": "http://existingpeer1.com:8080" + } + ] +} +``` + +From there, Tessera will connect to that peer and discover all the other PTM nodes in the network, connecting to each +of them in turn. + +!!! note + You may want to include multiple peers in the peer list in case any of them are offline/unreachable. + +### IP whitelisting + +The IP Whitelist that Tessera provides allows you restrict connections much like the `permissioned-nodes.json` file +does for Quorum. Only IP addresses/hostnames listed in your peers list will be allowed to connect to you. + +See the [Tessera configuration page](/Privacy/Tessera/Configuration/Configuration%20Overview#whitelist) for details on setting it up. + +In order to make sure the new node is accepted into the network: + +1. You will need to add the new peer to each of the existing nodes before communication is allowed. + Tessera provides a way to do this without needing to restart an already running node: + ```bash + $ java -jar tessera.jar admin -configfile /path/to/existing-node-config.json -addpeer http://newpeer.com:8080 + ``` + +2. The new peer can be started, setting the `peers` configuration to mirror the existing network. + e.g. if there are 3 existing nodes in the network, then the new nodes configuration will look like this: + ```json + { + "peers": [ + { + "url": "http://existingpeer1.com:8080" + }, + { + "url": "http://existingpeer2.com:8080" + }, + { + "url": "http://existingpeer3.com:8080" + } + ] + } + ``` + + The new node will allow incoming connections from the existing peers, and then existing peers will allow incoming + connections from the new peer! + +### Discovery + +Tessera discovery is very similar to the IP whitelist. The difference being that the IP whitelist blocks +communications between nodes, whereas disabling discovery only affects which public keys we keep track of. + +See the [Tessera configuration page](/Privacy/Tessera/Configuration/Configuration%20Overview#disabling-peer-discovery) for +details on setting it up. + +When discovery is disabled, Tessera will only allow keys that are owned by a node in its peer list to be available to +the users. This means that if any keys are found that are owned by a node NOT in our peer list, they are discarded and +private transactions cannot be sent to that public key. + +!!! note + This does not affect incoming transactions. Someone not in your peer list can still send transactions to your node, unless you also enable the IP Whitelist option. + + + +In order to make sure the new node is accepted into the network: + +1. You will need to add the new peer to each of the existing nodes before they will accept public keys that are linked +to the new peer. + Tessera provides a way to do this without needing to restart an already running node: + ```bash + $ java -jar tessera.jar admin -configfile /path/to/existing-node-config.json -addpeer http://newpeer.com:8080 + ``` + +2. The new peer can be started, setting the `peers` configuration to mirror the existing network. + e.g. if there are 3 existing nodes in the network, then the new nodes configuration will look like this: + ```json + { + "peers": [ + { + "url": "http://existingpeer1.com:8080" + }, + { + "url": "http://existingpeer2.com:8080" + }, + { + "url": "http://existingpeer3.com:8080" + } + ] + } + ``` + + The new node will now record public keys belonging to the existing peers, and then existing peers will record + public keys belonging to the new peer; this allows private transactions to be sent both directions! + + +## Examples + +For a walkthrough of some examples that put into action the above, check out [this guide](/How-To-Guides/add_node_examples)! \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index ef4211858..cf02102c0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -80,6 +80,9 @@ nav: - Getting Started: RemixPlugin/Getting started.md - Quorum Features: - DNS: Features/dns.md + - How-To Guides: + - Adding new nodes: How-To-Guides/adding_nodes.md + - Adding IBFT validators: How-To-Guides/add_ibft_validator.md - Product Roadmap: roadmap.md - FAQ: FAQ.md