Add insturctions to run a replicator on testnet (#4733)

book/src/SUMMARY.md

@@ -6,6 +6,7 @@
 
 - [Getting Started](getting-started.md)
   - [Testnet Participation](testnet-participation.md)
+  - [Testnet Replicator](testnet-replicator.md)
   - [Example: Web Wallet](webwallet.md)
 
 - [Programming Model](programs.md)

book/src/testnet-replicator.md

@@ -0,0 +1,154 @@
+## Testnet Replicator
+This document describes how to setup a replicator in the testnet
+
+Please note some of the information and instructions described here may change
+in future releases.
+
+### Overview
+Replicators are specialized light clients. They download a part of the
+ledger (a.k.a Segment) and store it. They earn rewards for storing segments.
+
+The testnet features a validator running at testnet.solana.com, which
+serves as the entrypoint to the cluster for your replicator node.
+
+Additionally there is a blockexplorer available at
+[http://testnet.solana.com/](http://testnet.solana.com/).
+
+The testnet is configured to reset the ledger daily, or sooner
+should the hourly automated cluster sanity test fail.
+
+### Machine Requirements
+Replicators don't need specialized hardware. Anything with more than
+128GB of disk space will be able to participate in the cluster as a replicator node.
+
+Currently the disk space requirements are very low but we expect them to change
+in the future.
+
+Prebuilt binaries are available for Linux x86_64 (Ubuntu 18.04 recommended),
+macOS, and Windows.
+
+#### Confirm The Testnet Is Reachable
+Before starting a replicator node, sanity check that the cluster is accessible
+to your machine by running some simple commands.  If any of the commands fail,
+please retry 5-10 minutes later to confirm the testnet is not just restarting
+itself before debugging further.
+
+Fetch the current transaction count over JSON RPC:
+```bash
+$ curl -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":1, "method":"getTransactionCount"}' http://testnet.solana.com:8899
+```
+
+Inspect the blockexplorer at [http://testnet.solana.com/](http://testnet.solana.com/) for activity.
+
+View the [metrics dashboard](
+https://metrics.solana.com:3000/d/testnet-beta/testnet-monitor-beta?var-testnet=testnet)
+for more detail on cluster activity.
+
+### Replicator Setup
+##### Obtaining The Software
+##### Bootstrap with `solana-install`
+
+The `solana-install` tool can be used to easily install and upgrade the cluster
+software.
+
+##### Linux and mac OS
+```bash
+$ export SOLANA_RELEASE=v0.16.0  # skip this line to install the latest release
+$ curl -sSf https://raw.githubusercontent.com/solana-labs/solana/v0.16.0/install/solana-install-init.sh | sh -s
+```
+
+Alternatively build the `solana-install` program from source and run the
+following command to obtain the same result:
+```bash
+$ solana-install init
+```
+
+##### Windows
+Download and install **solana-install-init** from
+[https://github.com/solana-labs/solana/releases/latest](https://github.com/solana-labs/solana/releases/latest)
+
+After a successful install, `solana-install update` may be used to
+easily update the software to a newer version at any time.
+
+##### Download Prebuilt Binaries
+If you would rather not use `solana-install` to manage the install, you can manually download and install the binaries.
+
+##### Linux
+Download the binaries by navigating to
+[https://github.com/solana-labs/solana/releases/latest](https://github.com/solana-labs/solana/releases/latest),
+download **solana-release-x86_64-unknown-linux-gnu.tar.bz2**, then extract the
+archive:
+```bash
+$ tar jxf solana-release-x86_64-unknown-linux-gnu.tar.bz2
+$ cd solana-release/
+$ export PATH=$PWD/bin:$PATH
+```
+##### mac OS
+Download the binaries by navigating to
+[https://github.com/solana-labs/solana/releases/latest](https://github.com/solana-labs/solana/releases/latest),
+download **solana-release-x86_64-apple-darwin.tar.bz2**, then extract the
+archive:
+```bash
+$ tar jxf solana-release-x86_64-apple-darwin.tar.bz2
+$ cd solana-release/
+$ export PATH=$PWD/bin:$PATH
+```
+##### Windows
+Download the binaries by navigating to
+[https://github.com/solana-labs/solana/releases/latest](https://github.com/solana-labs/solana/releases/latest),
+download **solana-release-x86_64-pc-windows-msvc.tar.bz2**, then extract it into a folder.
+It is a good idea to add this extracted folder to your windows PATH.
+
+### Starting The Replicator
+Try running following command to join the gossip network and view all the other nodes in the cluster:
+```bash
+$ solana-gossip --entrypoint testnet.solana.com:8001 spy
+# Press ^C to exit
+```
+
+Now configure the keypairs for your replicator by running:
+
+Navigate to the solana install location and open a cmd prompt
+```bash
+$ solana-keygen new -o replicator-keypair.json
+$ solana-keygen new -o storage-keypair.json
+```
+
+Use solana-keygen to show the public keys for each of the keypairs,
+they will be needed in the next step:
+- Windows
+```bash
+# The replicator's identity
+$ solana-keygen pubkey replicator-keypair.json
+$ solana-keygen pubkey storage-keypair.json
+```
+- Linux and mac OS
+```bash
+$ export REPLICATOR_IDENTITY=$(solana-keygen pubkey replicator-keypair.json)
+$ export STORAGE_IDENTITY=$(solana-keygen pubkey storage-keypair.json)
+
+```
+Then set up the storage accounts for your replicator by running:
+```bash
+$ solana-wallet --keypair replicator-keypair.json airdrop 100000
+$ solana-wallet --keypair replicator-keypair.json create-replicator-storage-account $REPLICATOR_IDENTITY $STORAGE_IDENTITY
+```
+Note: Every time the testnet restarts, run the wallet steps to setup the replicator accounts again.
+
+To start the replicator:
+```bash
+$ solana-replicator --entrypoint testnet.solana.com:8001 --identity replicator-keypair.json --storage-keypair storage-keypair.json --ledger replicator-ledger
+```
+
+### Verify Replicator Setup
+From another console, confirm the IP address and **identity pubkey** of your replicator is visible in the
+gossip network by running:
+```bash
+$ solana-gossip --entrypoint testnet.solana.com:8001 spy
+```
+
+Provide the **storage account pubkey** to the `solana-wallet show-storage-account` command to view
+the recent mining activity from your replicator:
+```bash
+$ solana-wallet --keypair storage-keypair.json show-storage-account $STORAGE_IDENTITY
+```

core/src/replicator.rs

@@ -303,7 +303,7 @@ impl Replicator {
         })
     }
 
-    pub fn run(&mut self, mining_pool_pubkey: Option<Pubkey>) {
+    pub fn run(&mut self, mining_pool_pubkey: Pubkey) {
         info!("waiting for ledger download");
         self.thread_handles.pop().unwrap().join().unwrap();
         self.encrypt_ledger()
@@ -330,9 +330,7 @@ impl Replicator {
                     }
                 };
             self.blockhash = storage_blockhash;
-            if let Some(mining_pool_pubkey) = mining_pool_pubkey {
-                self.redeem_rewards(&mining_pool_pubkey);
-            }
+            self.redeem_rewards(&mining_pool_pubkey);
         }
     }
 

genesis/src/main.rs

@@ -147,14 +147,6 @@ fn main() -> Result<(), Box<dyn error::Error>> {
                 .required(true)
                 .help("Path to file containing the bootstrap leader's storage keypair"),
         )
-        .arg(
-            Arg::with_name("storage_mining_pool_keypair_file")
-                .long("storage-mining-pool-keypair")
-                .value_name("KEYPAIR")
-                .takes_value(true)
-                .required(true)
-                .help("Path to file containing the storage mining pool storage keypair"),
-        )
         .arg(
             Arg::with_name("storage_mining_pool_lamports")
                 .long("storage-mining-pool-lamports")
@@ -263,9 +255,6 @@ fn main() -> Result<(), Box<dyn error::Error>> {
     let bootstrap_stake_keypair_file = matches.value_of("bootstrap_stake_keypair_file").unwrap();
     let bootstrap_storage_keypair_file =
         matches.value_of("bootstrap_storage_keypair_file").unwrap();
-    let storage_mining_pool_keypair = matches
-        .value_of("storage_mining_pool_keypair_file")
-        .unwrap();
     let mint_keypair_file = matches.value_of("mint_keypair_file").unwrap();
     let ledger_path = matches.value_of("ledger_path").unwrap();
     let lamports = value_t_or_exit!(matches, "lamports", u64);
@@ -278,7 +267,6 @@ fn main() -> Result<(), Box<dyn error::Error>> {
     let bootstrap_vote_keypair = read_keypair(bootstrap_vote_keypair_file)?;
     let bootstrap_stake_keypair = read_keypair(bootstrap_stake_keypair_file)?;
     let bootstrap_storage_keypair = read_keypair(bootstrap_storage_keypair_file)?;
-    let storage_mining_keypair = read_keypair(storage_mining_pool_keypair)?;
     let mint_keypair = read_keypair(mint_keypair_file)?;
 
     let (vote_account, vote_state) = vote_state::create_bootstrap_leader_account(
@@ -319,7 +307,9 @@ fn main() -> Result<(), Box<dyn error::Error>> {
                 ),
             ),
             (
-                storage_mining_keypair.pubkey(),
+                "StorageMiningPoo111111111111111111111111111"
+                    .parse()
+                    .unwrap(),
                 storage_contract::create_mining_pool_account(storage_pool_lamports),
             ),
         ])

multinode-demo/fullnode.sh

@@ -299,12 +299,6 @@ EOF
   default_arg --storage-keypair "$storage_keypair_path"
   default_arg --ledger "$ledger_config_dir"
 
-  storage_mining_pool_keypair_path="$SOLANA_CONFIG_DIR"/storage-mining-pool-keypair.json
-  if [[ -r $storage_mining_pool_keypair_path ]]; then
-    storage_mining_pool_pubkey=$($solana_keygen pubkey "$storage_mining_pool_keypair_path")
-    default_arg --mining-pool "$storage_mining_pool_pubkey"
-  fi
-
   rsync_entrypoint_url=$(rsync_url "$entrypoint")
 elif [[ $node_type = bootstrap_leader ]]; then
   if [[ ${#positional_args[@]} -ne 0 ]]; then

multinode-demo/setup.sh

@@ -13,14 +13,12 @@ $solana_keygen new -o "$SOLANA_CONFIG_DIR"/bootstrap-leader-keypair.json
 $solana_keygen new -o "$SOLANA_CONFIG_DIR"/bootstrap-leader-vote-keypair.json
 $solana_keygen new -o "$SOLANA_CONFIG_DIR"/bootstrap-leader-stake-keypair.json
 $solana_keygen new -o "$SOLANA_CONFIG_DIR"/bootstrap-leader-storage-keypair.json
-$solana_keygen new -o "$SOLANA_CONFIG_DIR"/storage-mining-pool-keypair.json
 
 args=("$@")
 default_arg --bootstrap-leader-keypair "$SOLANA_CONFIG_DIR"/bootstrap-leader-keypair.json
 default_arg --bootstrap-vote-keypair "$SOLANA_CONFIG_DIR"/bootstrap-leader-vote-keypair.json
 default_arg --bootstrap-stake-keypair "$SOLANA_CONFIG_DIR"/bootstrap-leader-stake-keypair.json
 default_arg --bootstrap-storage-keypair "$SOLANA_CONFIG_DIR"/bootstrap-leader-storage-keypair.json
-default_arg --storage-mining-pool-keypair "$SOLANA_CONFIG_DIR"/storage-mining-pool-keypair.json
 default_arg --ledger "$SOLANA_RSYNC_CONFIG_DIR"/ledger
 default_arg --mint "$SOLANA_CONFIG_DIR"/mint-keypair.json
 default_arg --lamports 100000000000000

replicator/src/main.rs

@@ -8,14 +8,6 @@ use std::net::SocketAddr;
 use std::process::exit;
 use std::sync::Arc;
 
-// Return an error if a pubkey cannot be parsed.
-fn is_pubkey(string: String) -> Result<(), String> {
-    match string.parse::<Pubkey>() {
-        Ok(_) => Ok(()),
-        Err(err) => Err(format!("{:?}", err)),
-    }
-}
-
 fn main() {
     solana_logger::setup();
 
@@ -30,14 +22,6 @@ fn main() {
                 .takes_value(true)
                 .help("File containing an identity (keypair)"),
         )
-        .arg(
-            Arg::with_name("storage_mining_pool_pubkey")
-                .long("mining-pool")
-                .value_name("PUBKEY_BASE58_STR")
-                .takes_value(true)
-                .validator(is_pubkey)
-                .help("The public key of the storage mining pool"),
-        )
         .arg(
             Arg::with_name("entrypoint")
                 .short("n")
@@ -86,9 +70,9 @@ fn main() {
         Keypair::new()
     };
 
-    let storage_mining_pool_pubkey = matches
-        .value_of("storage_mining_pool_pubkey")
-        .map(|value| value.parse::<Pubkey>().unwrap());
+    let storage_mining_pool_pubkey = "StorageMiningPoo111111111111111111111111111"
+        .parse::<Pubkey>()
+        .unwrap();
 
     let entrypoint_addr = matches
         .value_of("entrypoint")