add(book): Adds a section on Regtest usage to the Zebra book (#8526)

* Always set mempool.debug_enable_at_height to 0 on Regtest

* Adds a Regtest with Zebra page to the book

* Deduplicates calls to is_regtest() in start fn

* upgrade min rust version in zebrad

* Display full network name in logs except for Mainnet and default Testnet

* Add example logs/code to regtest section of the book

* apply some extra information

Co-authored-by: Arya <aryasolhi@gmail.com>

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>

book/src/SUMMARY.md

@@ -20,6 +20,7 @@
     - [Shielded Scanning gRPC Server](user/shielded-scan-grpc-server.md)
   - [Kibana blockchain explorer](user/elasticsearch.md)
   - [Forking the Zcash Testnet with Zebra](user/fork-zebra-testnet.md)
+  - [Regtest with Zebra](user/regtest.md)
   - [OpenAPI specification](user/openapi.md)
   - [Troubleshooting](user/troubleshooting.md)
 - [Developer Documentation](dev.md)

book/src/user/regtest.md

@@ -0,0 +1,101 @@
+# Regtest with Zebra
+
+The Regtest network in Zebra enables testing of custom functionalities in a private testnet environment with configurable network upgrade activation heights. It allows for starting an isolated node which won't connect to any peers and currently allows for committing blocks without validating their Proof of Work (in the future, it may use a very low target difficulty and easier Equihash parameters instead of skipping Proof of Work validation altogether).
+
+Zebra always activates the Canopy network upgrade at block height 1 due to limitations on its block construction.
+
+In order to use Regtest, Zebra must be configured to run on the Regtest network. The `[mining]` section is also necessary for mining blocks, and the `[rpc]` section is necessary for using the `send_raw_transaction` RPC method to mine non-coinbase transactions onto the chain.
+
+Relevant parts of the configuration file:
+
+```toml
+[mining]
+miner_address = 't27eWDgjFYJGVXmzrXeVjnb5J3uXDM9xH9v'
+    
+[network]
+network = "Regtest"
+
+# This section may be omitted when testing only Canopy
+[network.testnet_parameters.activation_heights]
+# Configured activation heights must be greater than or equal to 1, 
+# block height 0 is reserved for the Genesis network upgrade in Zebra
+NU5 = 1 
+
+# This section may be omitted if a persistent Regtest chain state is desired
+[state]
+ephemeral = true
+
+# This section may be omitted if it's not necessary to send transactions to Zebra's mempool
+[rpc]
+listen_addr = "0.0.0.0:18232"
+```
+
+Zebra should now include the Regtest network name in its logs, for example:
+
+```console
+2024-05-15T21:33:57.044156Z  INFO {zebrad="01633af" net="Regtest"}: zebrad::commands::start: initializing mempool
+```
+
+There are two ways to commit blocks to Zebra's state on Regtest:
+- Using the `getblocktemplate` and `submitblock` RPC methods directly
+- Using Zebra's experimental `internal-miner` feature
+
+## Using Zebra's Internal Miner
+
+Zebra can mine blocks on the Regtest network when compiled with the experimental `internal-miner` compilation feature and configured to enable to internal miner.
+
+Add `internal_miner = true` in the mining section of its configuration and compile Zebra with `cargo build --features "internal-miner"` (or `cargo run --features "internal-miner"` to compile and start Zebra) to use the internal miner with Regtest:
+
+```toml
+[mining]
+internal_miner = true
+```
+
+Zebra should now mine blocks on Regtest when it starts after a short delay (of around 30 seconds).
+
+To confirm that it's working, look for `successfully mined a new block` messages in the logs, or that the tip height is increasing.
+
+## Using RPC methods directly
+
+Blocks could also be mined outside of Zebra and submitted via Zebra's RPC methods. This requires enabling the RPC server in the configuration by providing a `listen_addr` field:
+
+```toml
+[rpc]
+listen_addr = "0.0.0.0:18232"
+```
+
+With Proof of Work disabled on Regtest, block templates can be converted directly into blocks with the `proposal_block_from_template()` function in the `zebra-chain` crate, serialized, hex-encoded, and then submitted via the `submitblock` RPC method.
+
+The `submitblock` RPC method should return `{ "result": null }` for successful block submissions.
+
+For example:
+
+```rust
+let client = RpcRequestClient::new(rpc_address);
+
+let block_template: GetBlockTemplate = client
+    .json_result_from_call("getblocktemplate", "[]".to_string())
+    .await
+    .expect("response should be success output with a serialized `GetBlockTemplate`");
+
+let network_upgrade = if block_template.height < NU5_ACTIVATION_HEIGHT {
+    NetworkUpgrade::Canopy
+} else {
+    NetworkUpgrade::Nu5
+};
+
+let block_data = hex::encode(
+    proposal_block_from_template(&block_template, TimeSource::default(), network_upgrade)?
+        .zcash_serialize_to_vec()?,
+);
+
+let submit_block_response = client
+    .text_from_call("submitblock", format!(r#"["{block_data}"]"#))
+    .await?;
+
+let was_submission_successful = submit_block_response.contains(r#""result":null"#);
+```
+
+See the `regtest_submit_blocks()` acceptance test as a more detailed example for using Zebra's RPC methods to submit blocks on Regtest.
+
+When Proof of Work validation is enabled for Regtest with a low target difficulty and easy Equihash parameters, Zebra may have a `network.testnet_parameters.disable_pow` field in its configuration so that this would continue working.

zebrad/Cargo.toml

@@ -19,7 +19,7 @@ edition = "2021"
 
 # Zebra is only supported on the latest stable Rust version. See the README for details.
 # Any Zebra release can break compatibility with older Rust versions.
-rust-version = "1.73"
+rust-version = "1.76"
 
 # Settings that impact runtime behaviour
 

zebrad/src/application.rs

@@ -455,7 +455,11 @@ impl Application for ZebradApp {
         // Activate the global span, so it's visible when we load the other
         // components. Space is at a premium here, so we use an empty message,
         // short commit hash, and the unique part of the network name.
-        let net = &config.network.network.to_string()[..4];
+        let net = config.network.network.to_string();
+        let net = match net.as_str() {
+            default_net_name @ ("Testnet" | "Mainnet") => &default_net_name[..4],
+            other_net_name => other_net_name,
+        };
         let global_span = if let Some(git_commit) = ZebradApp::git_commit() {
             error_span!("", zebrad = git_commit, net)
         } else {

zebrad/src/commands/start.rs

@@ -78,6 +78,8 @@
 //!
 //! Some of the diagnostic features are optional, and need to be enabled at compile-time.
 
+use std::sync::Arc;
+
 use abscissa_core::{config, Command, FrameworkError};
 use color_eyre::eyre::{eyre, Report};
 use futures::FutureExt;
@@ -113,6 +115,19 @@ pub struct StartCmd {
 impl StartCmd {
     async fn start(&self) -> Result<(), Report> {
         let config = APPLICATION.config();
+        let is_regtest = config.network.network.is_regtest();
+
+        let config = if is_regtest {
+            Arc::new(ZebradConfig {
+                mempool: mempool::Config {
+                    debug_enable_at_height: Some(0),
+                    ..config.mempool
+                },
+                ..Arc::unwrap_or_clone(config)
+            })
+        } else {
+            config
+        };
 
         info!("initializing node state");
         let (_, max_checkpoint_height) = zebra_consensus::router::init_checkpoint_list(
@@ -301,7 +316,7 @@ impl StartCmd {
         );
 
         info!("spawning syncer task");
-        let syncer_task_handle = if config.network.network.is_regtest() {
+        let syncer_task_handle = if is_regtest {
             if !syncer
                 .state_contains(config.network.network.genesis_hash())
                 .await?