zebra/zebrad/tests/common/lightwalletd/wallet_grpc.rs

58 lines
2.2 KiB
Rust
Raw Normal View History

add(test): Integration test to send transactions using lightwalletd (#4068) * Export the `zebra_state::Config::db_path` method Make it easier for tests to discover the sub-directory used to store the chain state data. * Generate code for interfacing with lightwalletd Use the `tonic-build` crate to generate Rust code for communicating with lightwalletd using gRPC. The `*.proto` files were obtained from the Zcash lightwalletd repository. * Use `block::Height` instead of `Height` Import the `block` instead to make it slightly clearer. * Add helper function to remove a file if it exists Try to remove it and ignore an error if it says that the file doesn't exist. This will be used later to remove the lock file from a copied chain state directory. * Add helper function to copy chain state dirs Copy an existing chain state directory into a new temporary directory. * Add a `BoxStateService` type alias Make it easier to write and read a boxed version of a state service. * Add a helper function to start the state service Make it easier to specify the state service to use an existing state cache directory. * Import `eyre!` macro at the module level Allow it to be used in different places without having to repeat the imports. * Add `load_tip_height_from_state_directory` helper A function to discover the current chain tip height stored in a state cache. * Add helper function to prepare partial sync. state Loads a partially synchronized cached state directory into a temporary directory that can be used by a zebrad instance, and also returns the chain tip block height of that state. * Add `perform_full_sync_starting_from` helper Runs a zebrad with an existing partially synchronized state, and finishes synchronizing it to the network chain tip. * Add function to load transactions from a block Use a provided state service to load all transactions from a block at a specified height. The state service is a generic type parameter, because `zebra_state::service::ReadStateService` is not exported publicly. Using a generic type parameter also allows the service to be wrapped in layers if needed in the future. * Add `load_transactions_from_block_after` helper A function to load transactions from a block stored in a cached state directory. The cached state must be synchronized to a chain tip higher than the requested height. * Add helper function to load some test transactions Given a partially synchronized chain state, it will extend that chain by performing a full synchronization, and obtain some transactions from one of the newly added blocks. * Update `spawn_zebrad_for_rpc_without_initial_peers` Wait until the mempool is activated. * Add method to start lightwalletd with RPC server Returns the lightwalletd instance and the port that it's listening for RPC connections. The instance can reuse an existing cached lightwalletd state if the `LIGHTWALLETD_DATA_DIR` environment variable is set. * Add a `LightwalletdRpcClient` type alias To make it easier to identify the type generated from the Protobuf files. * Add helper function to connect to lightwalletd Prepare an RPC client to send requests to a lightwalletd instance. * Add a `prepare_send_transaction_request` helper Creates a request message for lightwalletd to send a transaction. * Add test to send transactions using lightwalletd Obtain some valid transactions from future blocks and try to send them to a lightwalletd instance connected to a zebrad instance that hasn't seen those transactions yet. The transactions should be successfully queued in Zebra's mempool. * Make `zebra_directory` parameter generic Allow using a `TempDir` or a `PathBuf`. * Move lightwalletd protobuf files Place them closer to the module directory, so that it's clearer that they specify the RPC protocol for lightwalletd, and not Zebra itself. * Don't use coinbase transactions in the test Coinbase transactions are rejected by the mempool. * Don't remove state lock file It is removed automatically by Zebra when it shuts down, so if it exists it should be reported as a bug. * Force mempool to be enabled in Zebrad instance Speed up the initialization of the Zebrad instance used for lightwalletd to connect to. * Refactor to create `LIGHTWALLETD_DATA_DIR_VAR` Document how the environment variable can be used to speed up the test. * Check for process errors in spawned Zebra instance Enable checking for known process failure messages. * Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant Document why it exists and how the choice of the value affects the test. * Add `LIGHTWALLETD_TEST_TIMEOUT` constant And use it for the Zebrad and the Lightwalletd instances used in the send transaction integration test. * Check `lightwalletd` process for errors Enable checking the lightwalletd process for known failure messages. * Update `tonic` and `prost` dependencies Use the latest version and fix CI failures because `rustfmt` isn't installed in the build environment. * Create `send_transaction_test` module Move the send transaction using lightwalletd test and its helper functions into a new module. * Move `LIGHTWALLETD_TEST_TIMEOUT` constant Place it in the parent `lightwalletd` module. * Move gRPC helper functions and types to `rpc` mod. Make them more accessible so that they can be used by other tests. * Create a `cached_state` module Move the test utility functions related to using a cached Zebra state into the module. * Move `perform_full_sync_starting_from` to `sync` Keep to closer to the synchronization utility functions. * Move Zebra cached state path variable constant Place it in the `cached_state` module. * Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set Make it part of the set of tests ignored as a whole if no lightwalletd tests should be executed. * Move `spawn_zebrad_for_rpc_without_initial_peers` Place it in the `launch` sub-module. * Rename `rpc` module into `wallet_grpc` Avoid any potential misunderstandings when the name is seen out of context. * Allow duplicate `heck` dependency At least until `structopt` is updated or `zebra-utils` is updated to use `clap` 3. * Fix a deny.toml typo * fix(build): CMake is required by `prost` crate Co-authored-by: teor <teor@riseup.net> Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 16:06:11 -07:00
//! Lightwalletd gRPC interface and utility functions.
use std::{env, net::SocketAddr, path::PathBuf};
add(test): Integration test to send transactions using lightwalletd (#4068) * Export the `zebra_state::Config::db_path` method Make it easier for tests to discover the sub-directory used to store the chain state data. * Generate code for interfacing with lightwalletd Use the `tonic-build` crate to generate Rust code for communicating with lightwalletd using gRPC. The `*.proto` files were obtained from the Zcash lightwalletd repository. * Use `block::Height` instead of `Height` Import the `block` instead to make it slightly clearer. * Add helper function to remove a file if it exists Try to remove it and ignore an error if it says that the file doesn't exist. This will be used later to remove the lock file from a copied chain state directory. * Add helper function to copy chain state dirs Copy an existing chain state directory into a new temporary directory. * Add a `BoxStateService` type alias Make it easier to write and read a boxed version of a state service. * Add a helper function to start the state service Make it easier to specify the state service to use an existing state cache directory. * Import `eyre!` macro at the module level Allow it to be used in different places without having to repeat the imports. * Add `load_tip_height_from_state_directory` helper A function to discover the current chain tip height stored in a state cache. * Add helper function to prepare partial sync. state Loads a partially synchronized cached state directory into a temporary directory that can be used by a zebrad instance, and also returns the chain tip block height of that state. * Add `perform_full_sync_starting_from` helper Runs a zebrad with an existing partially synchronized state, and finishes synchronizing it to the network chain tip. * Add function to load transactions from a block Use a provided state service to load all transactions from a block at a specified height. The state service is a generic type parameter, because `zebra_state::service::ReadStateService` is not exported publicly. Using a generic type parameter also allows the service to be wrapped in layers if needed in the future. * Add `load_transactions_from_block_after` helper A function to load transactions from a block stored in a cached state directory. The cached state must be synchronized to a chain tip higher than the requested height. * Add helper function to load some test transactions Given a partially synchronized chain state, it will extend that chain by performing a full synchronization, and obtain some transactions from one of the newly added blocks. * Update `spawn_zebrad_for_rpc_without_initial_peers` Wait until the mempool is activated. * Add method to start lightwalletd with RPC server Returns the lightwalletd instance and the port that it's listening for RPC connections. The instance can reuse an existing cached lightwalletd state if the `LIGHTWALLETD_DATA_DIR` environment variable is set. * Add a `LightwalletdRpcClient` type alias To make it easier to identify the type generated from the Protobuf files. * Add helper function to connect to lightwalletd Prepare an RPC client to send requests to a lightwalletd instance. * Add a `prepare_send_transaction_request` helper Creates a request message for lightwalletd to send a transaction. * Add test to send transactions using lightwalletd Obtain some valid transactions from future blocks and try to send them to a lightwalletd instance connected to a zebrad instance that hasn't seen those transactions yet. The transactions should be successfully queued in Zebra's mempool. * Make `zebra_directory` parameter generic Allow using a `TempDir` or a `PathBuf`. * Move lightwalletd protobuf files Place them closer to the module directory, so that it's clearer that they specify the RPC protocol for lightwalletd, and not Zebra itself. * Don't use coinbase transactions in the test Coinbase transactions are rejected by the mempool. * Don't remove state lock file It is removed automatically by Zebra when it shuts down, so if it exists it should be reported as a bug. * Force mempool to be enabled in Zebrad instance Speed up the initialization of the Zebrad instance used for lightwalletd to connect to. * Refactor to create `LIGHTWALLETD_DATA_DIR_VAR` Document how the environment variable can be used to speed up the test. * Check for process errors in spawned Zebra instance Enable checking for known process failure messages. * Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant Document why it exists and how the choice of the value affects the test. * Add `LIGHTWALLETD_TEST_TIMEOUT` constant And use it for the Zebrad and the Lightwalletd instances used in the send transaction integration test. * Check `lightwalletd` process for errors Enable checking the lightwalletd process for known failure messages. * Update `tonic` and `prost` dependencies Use the latest version and fix CI failures because `rustfmt` isn't installed in the build environment. * Create `send_transaction_test` module Move the send transaction using lightwalletd test and its helper functions into a new module. * Move `LIGHTWALLETD_TEST_TIMEOUT` constant Place it in the parent `lightwalletd` module. * Move gRPC helper functions and types to `rpc` mod. Make them more accessible so that they can be used by other tests. * Create a `cached_state` module Move the test utility functions related to using a cached Zebra state into the module. * Move `perform_full_sync_starting_from` to `sync` Keep to closer to the synchronization utility functions. * Move Zebra cached state path variable constant Place it in the `cached_state` module. * Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set Make it part of the set of tests ignored as a whole if no lightwalletd tests should be executed. * Move `spawn_zebrad_for_rpc_without_initial_peers` Place it in the `launch` sub-module. * Rename `rpc` module into `wallet_grpc` Avoid any potential misunderstandings when the name is seen out of context. * Allow duplicate `heck` dependency At least until `structopt` is updated or `zebra-utils` is updated to use `clap` 3. * Fix a deny.toml typo * fix(build): CMake is required by `prost` crate Co-authored-by: teor <teor@riseup.net> Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 16:06:11 -07:00
use tempfile::TempDir;
use zebra_test::{args, net::random_known_port, prelude::*};
use crate::common::{config::testdir, lightwalletd::LightWalletdTestDirExt};
add(test): Integration test to send transactions using lightwalletd (#4068) * Export the `zebra_state::Config::db_path` method Make it easier for tests to discover the sub-directory used to store the chain state data. * Generate code for interfacing with lightwalletd Use the `tonic-build` crate to generate Rust code for communicating with lightwalletd using gRPC. The `*.proto` files were obtained from the Zcash lightwalletd repository. * Use `block::Height` instead of `Height` Import the `block` instead to make it slightly clearer. * Add helper function to remove a file if it exists Try to remove it and ignore an error if it says that the file doesn't exist. This will be used later to remove the lock file from a copied chain state directory. * Add helper function to copy chain state dirs Copy an existing chain state directory into a new temporary directory. * Add a `BoxStateService` type alias Make it easier to write and read a boxed version of a state service. * Add a helper function to start the state service Make it easier to specify the state service to use an existing state cache directory. * Import `eyre!` macro at the module level Allow it to be used in different places without having to repeat the imports. * Add `load_tip_height_from_state_directory` helper A function to discover the current chain tip height stored in a state cache. * Add helper function to prepare partial sync. state Loads a partially synchronized cached state directory into a temporary directory that can be used by a zebrad instance, and also returns the chain tip block height of that state. * Add `perform_full_sync_starting_from` helper Runs a zebrad with an existing partially synchronized state, and finishes synchronizing it to the network chain tip. * Add function to load transactions from a block Use a provided state service to load all transactions from a block at a specified height. The state service is a generic type parameter, because `zebra_state::service::ReadStateService` is not exported publicly. Using a generic type parameter also allows the service to be wrapped in layers if needed in the future. * Add `load_transactions_from_block_after` helper A function to load transactions from a block stored in a cached state directory. The cached state must be synchronized to a chain tip higher than the requested height. * Add helper function to load some test transactions Given a partially synchronized chain state, it will extend that chain by performing a full synchronization, and obtain some transactions from one of the newly added blocks. * Update `spawn_zebrad_for_rpc_without_initial_peers` Wait until the mempool is activated. * Add method to start lightwalletd with RPC server Returns the lightwalletd instance and the port that it's listening for RPC connections. The instance can reuse an existing cached lightwalletd state if the `LIGHTWALLETD_DATA_DIR` environment variable is set. * Add a `LightwalletdRpcClient` type alias To make it easier to identify the type generated from the Protobuf files. * Add helper function to connect to lightwalletd Prepare an RPC client to send requests to a lightwalletd instance. * Add a `prepare_send_transaction_request` helper Creates a request message for lightwalletd to send a transaction. * Add test to send transactions using lightwalletd Obtain some valid transactions from future blocks and try to send them to a lightwalletd instance connected to a zebrad instance that hasn't seen those transactions yet. The transactions should be successfully queued in Zebra's mempool. * Make `zebra_directory` parameter generic Allow using a `TempDir` or a `PathBuf`. * Move lightwalletd protobuf files Place them closer to the module directory, so that it's clearer that they specify the RPC protocol for lightwalletd, and not Zebra itself. * Don't use coinbase transactions in the test Coinbase transactions are rejected by the mempool. * Don't remove state lock file It is removed automatically by Zebra when it shuts down, so if it exists it should be reported as a bug. * Force mempool to be enabled in Zebrad instance Speed up the initialization of the Zebrad instance used for lightwalletd to connect to. * Refactor to create `LIGHTWALLETD_DATA_DIR_VAR` Document how the environment variable can be used to speed up the test. * Check for process errors in spawned Zebra instance Enable checking for known process failure messages. * Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant Document why it exists and how the choice of the value affects the test. * Add `LIGHTWALLETD_TEST_TIMEOUT` constant And use it for the Zebrad and the Lightwalletd instances used in the send transaction integration test. * Check `lightwalletd` process for errors Enable checking the lightwalletd process for known failure messages. * Update `tonic` and `prost` dependencies Use the latest version and fix CI failures because `rustfmt` isn't installed in the build environment. * Create `send_transaction_test` module Move the send transaction using lightwalletd test and its helper functions into a new module. * Move `LIGHTWALLETD_TEST_TIMEOUT` constant Place it in the parent `lightwalletd` module. * Move gRPC helper functions and types to `rpc` mod. Make them more accessible so that they can be used by other tests. * Create a `cached_state` module Move the test utility functions related to using a cached Zebra state into the module. * Move `perform_full_sync_starting_from` to `sync` Keep to closer to the synchronization utility functions. * Move Zebra cached state path variable constant Place it in the `cached_state` module. * Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set Make it part of the set of tests ignored as a whole if no lightwalletd tests should be executed. * Move `spawn_zebrad_for_rpc_without_initial_peers` Place it in the `launch` sub-module. * Rename `rpc` module into `wallet_grpc` Avoid any potential misunderstandings when the name is seen out of context. * Allow duplicate `heck` dependency At least until `structopt` is updated or `zebra-utils` is updated to use `clap` 3. * Fix a deny.toml typo * fix(build): CMake is required by `prost` crate Co-authored-by: teor <teor@riseup.net> Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 16:06:11 -07:00
use super::LightwalletdTestType;
add(test): Integration test to send transactions using lightwalletd (#4068) * Export the `zebra_state::Config::db_path` method Make it easier for tests to discover the sub-directory used to store the chain state data. * Generate code for interfacing with lightwalletd Use the `tonic-build` crate to generate Rust code for communicating with lightwalletd using gRPC. The `*.proto` files were obtained from the Zcash lightwalletd repository. * Use `block::Height` instead of `Height` Import the `block` instead to make it slightly clearer. * Add helper function to remove a file if it exists Try to remove it and ignore an error if it says that the file doesn't exist. This will be used later to remove the lock file from a copied chain state directory. * Add helper function to copy chain state dirs Copy an existing chain state directory into a new temporary directory. * Add a `BoxStateService` type alias Make it easier to write and read a boxed version of a state service. * Add a helper function to start the state service Make it easier to specify the state service to use an existing state cache directory. * Import `eyre!` macro at the module level Allow it to be used in different places without having to repeat the imports. * Add `load_tip_height_from_state_directory` helper A function to discover the current chain tip height stored in a state cache. * Add helper function to prepare partial sync. state Loads a partially synchronized cached state directory into a temporary directory that can be used by a zebrad instance, and also returns the chain tip block height of that state. * Add `perform_full_sync_starting_from` helper Runs a zebrad with an existing partially synchronized state, and finishes synchronizing it to the network chain tip. * Add function to load transactions from a block Use a provided state service to load all transactions from a block at a specified height. The state service is a generic type parameter, because `zebra_state::service::ReadStateService` is not exported publicly. Using a generic type parameter also allows the service to be wrapped in layers if needed in the future. * Add `load_transactions_from_block_after` helper A function to load transactions from a block stored in a cached state directory. The cached state must be synchronized to a chain tip higher than the requested height. * Add helper function to load some test transactions Given a partially synchronized chain state, it will extend that chain by performing a full synchronization, and obtain some transactions from one of the newly added blocks. * Update `spawn_zebrad_for_rpc_without_initial_peers` Wait until the mempool is activated. * Add method to start lightwalletd with RPC server Returns the lightwalletd instance and the port that it's listening for RPC connections. The instance can reuse an existing cached lightwalletd state if the `LIGHTWALLETD_DATA_DIR` environment variable is set. * Add a `LightwalletdRpcClient` type alias To make it easier to identify the type generated from the Protobuf files. * Add helper function to connect to lightwalletd Prepare an RPC client to send requests to a lightwalletd instance. * Add a `prepare_send_transaction_request` helper Creates a request message for lightwalletd to send a transaction. * Add test to send transactions using lightwalletd Obtain some valid transactions from future blocks and try to send them to a lightwalletd instance connected to a zebrad instance that hasn't seen those transactions yet. The transactions should be successfully queued in Zebra's mempool. * Make `zebra_directory` parameter generic Allow using a `TempDir` or a `PathBuf`. * Move lightwalletd protobuf files Place them closer to the module directory, so that it's clearer that they specify the RPC protocol for lightwalletd, and not Zebra itself. * Don't use coinbase transactions in the test Coinbase transactions are rejected by the mempool. * Don't remove state lock file It is removed automatically by Zebra when it shuts down, so if it exists it should be reported as a bug. * Force mempool to be enabled in Zebrad instance Speed up the initialization of the Zebrad instance used for lightwalletd to connect to. * Refactor to create `LIGHTWALLETD_DATA_DIR_VAR` Document how the environment variable can be used to speed up the test. * Check for process errors in spawned Zebra instance Enable checking for known process failure messages. * Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant Document why it exists and how the choice of the value affects the test. * Add `LIGHTWALLETD_TEST_TIMEOUT` constant And use it for the Zebrad and the Lightwalletd instances used in the send transaction integration test. * Check `lightwalletd` process for errors Enable checking the lightwalletd process for known failure messages. * Update `tonic` and `prost` dependencies Use the latest version and fix CI failures because `rustfmt` isn't installed in the build environment. * Create `send_transaction_test` module Move the send transaction using lightwalletd test and its helper functions into a new module. * Move `LIGHTWALLETD_TEST_TIMEOUT` constant Place it in the parent `lightwalletd` module. * Move gRPC helper functions and types to `rpc` mod. Make them more accessible so that they can be used by other tests. * Create a `cached_state` module Move the test utility functions related to using a cached Zebra state into the module. * Move `perform_full_sync_starting_from` to `sync` Keep to closer to the synchronization utility functions. * Move Zebra cached state path variable constant Place it in the `cached_state` module. * Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set Make it part of the set of tests ignored as a whole if no lightwalletd tests should be executed. * Move `spawn_zebrad_for_rpc_without_initial_peers` Place it in the `launch` sub-module. * Rename `rpc` module into `wallet_grpc` Avoid any potential misunderstandings when the name is seen out of context. * Allow duplicate `heck` dependency At least until `structopt` is updated or `zebra-utils` is updated to use `clap` 3. * Fix a deny.toml typo * fix(build): CMake is required by `prost` crate Co-authored-by: teor <teor@riseup.net> Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 16:06:11 -07:00
tonic::include_proto!("cash.z.wallet.sdk.rpc");
add(test): Integration test to send transactions using lightwalletd (#4068) * Export the `zebra_state::Config::db_path` method Make it easier for tests to discover the sub-directory used to store the chain state data. * Generate code for interfacing with lightwalletd Use the `tonic-build` crate to generate Rust code for communicating with lightwalletd using gRPC. The `*.proto` files were obtained from the Zcash lightwalletd repository. * Use `block::Height` instead of `Height` Import the `block` instead to make it slightly clearer. * Add helper function to remove a file if it exists Try to remove it and ignore an error if it says that the file doesn't exist. This will be used later to remove the lock file from a copied chain state directory. * Add helper function to copy chain state dirs Copy an existing chain state directory into a new temporary directory. * Add a `BoxStateService` type alias Make it easier to write and read a boxed version of a state service. * Add a helper function to start the state service Make it easier to specify the state service to use an existing state cache directory. * Import `eyre!` macro at the module level Allow it to be used in different places without having to repeat the imports. * Add `load_tip_height_from_state_directory` helper A function to discover the current chain tip height stored in a state cache. * Add helper function to prepare partial sync. state Loads a partially synchronized cached state directory into a temporary directory that can be used by a zebrad instance, and also returns the chain tip block height of that state. * Add `perform_full_sync_starting_from` helper Runs a zebrad with an existing partially synchronized state, and finishes synchronizing it to the network chain tip. * Add function to load transactions from a block Use a provided state service to load all transactions from a block at a specified height. The state service is a generic type parameter, because `zebra_state::service::ReadStateService` is not exported publicly. Using a generic type parameter also allows the service to be wrapped in layers if needed in the future. * Add `load_transactions_from_block_after` helper A function to load transactions from a block stored in a cached state directory. The cached state must be synchronized to a chain tip higher than the requested height. * Add helper function to load some test transactions Given a partially synchronized chain state, it will extend that chain by performing a full synchronization, and obtain some transactions from one of the newly added blocks. * Update `spawn_zebrad_for_rpc_without_initial_peers` Wait until the mempool is activated. * Add method to start lightwalletd with RPC server Returns the lightwalletd instance and the port that it's listening for RPC connections. The instance can reuse an existing cached lightwalletd state if the `LIGHTWALLETD_DATA_DIR` environment variable is set. * Add a `LightwalletdRpcClient` type alias To make it easier to identify the type generated from the Protobuf files. * Add helper function to connect to lightwalletd Prepare an RPC client to send requests to a lightwalletd instance. * Add a `prepare_send_transaction_request` helper Creates a request message for lightwalletd to send a transaction. * Add test to send transactions using lightwalletd Obtain some valid transactions from future blocks and try to send them to a lightwalletd instance connected to a zebrad instance that hasn't seen those transactions yet. The transactions should be successfully queued in Zebra's mempool. * Make `zebra_directory` parameter generic Allow using a `TempDir` or a `PathBuf`. * Move lightwalletd protobuf files Place them closer to the module directory, so that it's clearer that they specify the RPC protocol for lightwalletd, and not Zebra itself. * Don't use coinbase transactions in the test Coinbase transactions are rejected by the mempool. * Don't remove state lock file It is removed automatically by Zebra when it shuts down, so if it exists it should be reported as a bug. * Force mempool to be enabled in Zebrad instance Speed up the initialization of the Zebrad instance used for lightwalletd to connect to. * Refactor to create `LIGHTWALLETD_DATA_DIR_VAR` Document how the environment variable can be used to speed up the test. * Check for process errors in spawned Zebra instance Enable checking for known process failure messages. * Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant Document why it exists and how the choice of the value affects the test. * Add `LIGHTWALLETD_TEST_TIMEOUT` constant And use it for the Zebrad and the Lightwalletd instances used in the send transaction integration test. * Check `lightwalletd` process for errors Enable checking the lightwalletd process for known failure messages. * Update `tonic` and `prost` dependencies Use the latest version and fix CI failures because `rustfmt` isn't installed in the build environment. * Create `send_transaction_test` module Move the send transaction using lightwalletd test and its helper functions into a new module. * Move `LIGHTWALLETD_TEST_TIMEOUT` constant Place it in the parent `lightwalletd` module. * Move gRPC helper functions and types to `rpc` mod. Make them more accessible so that they can be used by other tests. * Create a `cached_state` module Move the test utility functions related to using a cached Zebra state into the module. * Move `perform_full_sync_starting_from` to `sync` Keep to closer to the synchronization utility functions. * Move Zebra cached state path variable constant Place it in the `cached_state` module. * Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set Make it part of the set of tests ignored as a whole if no lightwalletd tests should be executed. * Move `spawn_zebrad_for_rpc_without_initial_peers` Place it in the `launch` sub-module. * Rename `rpc` module into `wallet_grpc` Avoid any potential misunderstandings when the name is seen out of context. * Allow duplicate `heck` dependency At least until `structopt` is updated or `zebra-utils` is updated to use `clap` 3. * Fix a deny.toml typo * fix(build): CMake is required by `prost` crate Co-authored-by: teor <teor@riseup.net> Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 16:06:11 -07:00
/// Type alias for the RPC client to communicate with a lightwalletd instance.
pub type LightwalletdRpcClient =
compact_tx_streamer_client::CompactTxStreamerClient<tonic::transport::Channel>;
/// Start a lightwalletd instance with its RPC server functionality enabled.
///
/// Returns the lightwalletd instance and the port number that it is listening for RPC connections.
pub fn spawn_lightwalletd_with_rpc_server(
zebrad_rpc_address: SocketAddr,
lightwalletd_state_path: Option<PathBuf>,
test_type: LightwalletdTestType,
wait_for_blocks: bool,
add(test): Integration test to send transactions using lightwalletd (#4068) * Export the `zebra_state::Config::db_path` method Make it easier for tests to discover the sub-directory used to store the chain state data. * Generate code for interfacing with lightwalletd Use the `tonic-build` crate to generate Rust code for communicating with lightwalletd using gRPC. The `*.proto` files were obtained from the Zcash lightwalletd repository. * Use `block::Height` instead of `Height` Import the `block` instead to make it slightly clearer. * Add helper function to remove a file if it exists Try to remove it and ignore an error if it says that the file doesn't exist. This will be used later to remove the lock file from a copied chain state directory. * Add helper function to copy chain state dirs Copy an existing chain state directory into a new temporary directory. * Add a `BoxStateService` type alias Make it easier to write and read a boxed version of a state service. * Add a helper function to start the state service Make it easier to specify the state service to use an existing state cache directory. * Import `eyre!` macro at the module level Allow it to be used in different places without having to repeat the imports. * Add `load_tip_height_from_state_directory` helper A function to discover the current chain tip height stored in a state cache. * Add helper function to prepare partial sync. state Loads a partially synchronized cached state directory into a temporary directory that can be used by a zebrad instance, and also returns the chain tip block height of that state. * Add `perform_full_sync_starting_from` helper Runs a zebrad with an existing partially synchronized state, and finishes synchronizing it to the network chain tip. * Add function to load transactions from a block Use a provided state service to load all transactions from a block at a specified height. The state service is a generic type parameter, because `zebra_state::service::ReadStateService` is not exported publicly. Using a generic type parameter also allows the service to be wrapped in layers if needed in the future. * Add `load_transactions_from_block_after` helper A function to load transactions from a block stored in a cached state directory. The cached state must be synchronized to a chain tip higher than the requested height. * Add helper function to load some test transactions Given a partially synchronized chain state, it will extend that chain by performing a full synchronization, and obtain some transactions from one of the newly added blocks. * Update `spawn_zebrad_for_rpc_without_initial_peers` Wait until the mempool is activated. * Add method to start lightwalletd with RPC server Returns the lightwalletd instance and the port that it's listening for RPC connections. The instance can reuse an existing cached lightwalletd state if the `LIGHTWALLETD_DATA_DIR` environment variable is set. * Add a `LightwalletdRpcClient` type alias To make it easier to identify the type generated from the Protobuf files. * Add helper function to connect to lightwalletd Prepare an RPC client to send requests to a lightwalletd instance. * Add a `prepare_send_transaction_request` helper Creates a request message for lightwalletd to send a transaction. * Add test to send transactions using lightwalletd Obtain some valid transactions from future blocks and try to send them to a lightwalletd instance connected to a zebrad instance that hasn't seen those transactions yet. The transactions should be successfully queued in Zebra's mempool. * Make `zebra_directory` parameter generic Allow using a `TempDir` or a `PathBuf`. * Move lightwalletd protobuf files Place them closer to the module directory, so that it's clearer that they specify the RPC protocol for lightwalletd, and not Zebra itself. * Don't use coinbase transactions in the test Coinbase transactions are rejected by the mempool. * Don't remove state lock file It is removed automatically by Zebra when it shuts down, so if it exists it should be reported as a bug. * Force mempool to be enabled in Zebrad instance Speed up the initialization of the Zebrad instance used for lightwalletd to connect to. * Refactor to create `LIGHTWALLETD_DATA_DIR_VAR` Document how the environment variable can be used to speed up the test. * Check for process errors in spawned Zebra instance Enable checking for known process failure messages. * Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant Document why it exists and how the choice of the value affects the test. * Add `LIGHTWALLETD_TEST_TIMEOUT` constant And use it for the Zebrad and the Lightwalletd instances used in the send transaction integration test. * Check `lightwalletd` process for errors Enable checking the lightwalletd process for known failure messages. * Update `tonic` and `prost` dependencies Use the latest version and fix CI failures because `rustfmt` isn't installed in the build environment. * Create `send_transaction_test` module Move the send transaction using lightwalletd test and its helper functions into a new module. * Move `LIGHTWALLETD_TEST_TIMEOUT` constant Place it in the parent `lightwalletd` module. * Move gRPC helper functions and types to `rpc` mod. Make them more accessible so that they can be used by other tests. * Create a `cached_state` module Move the test utility functions related to using a cached Zebra state into the module. * Move `perform_full_sync_starting_from` to `sync` Keep to closer to the synchronization utility functions. * Move Zebra cached state path variable constant Place it in the `cached_state` module. * Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set Make it part of the set of tests ignored as a whole if no lightwalletd tests should be executed. * Move `spawn_zebrad_for_rpc_without_initial_peers` Place it in the `launch` sub-module. * Rename `rpc` module into `wallet_grpc` Avoid any potential misunderstandings when the name is seen out of context. * Allow duplicate `heck` dependency At least until `structopt` is updated or `zebra-utils` is updated to use `clap` 3. * Fix a deny.toml typo * fix(build): CMake is required by `prost` crate Co-authored-by: teor <teor@riseup.net> Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 16:06:11 -07:00
) -> Result<(TestChild<TempDir>, u16)> {
let lightwalletd_dir = testdir()?.with_lightwalletd_config(zebrad_rpc_address)?;
let lightwalletd_rpc_port = random_known_port();
let lightwalletd_rpc_address = format!("127.0.0.1:{lightwalletd_rpc_port}");
let arguments = args!["--grpc-bind-addr": lightwalletd_rpc_address];
add(test): Integration test to send transactions using lightwalletd (#4068) * Export the `zebra_state::Config::db_path` method Make it easier for tests to discover the sub-directory used to store the chain state data. * Generate code for interfacing with lightwalletd Use the `tonic-build` crate to generate Rust code for communicating with lightwalletd using gRPC. The `*.proto` files were obtained from the Zcash lightwalletd repository. * Use `block::Height` instead of `Height` Import the `block` instead to make it slightly clearer. * Add helper function to remove a file if it exists Try to remove it and ignore an error if it says that the file doesn't exist. This will be used later to remove the lock file from a copied chain state directory. * Add helper function to copy chain state dirs Copy an existing chain state directory into a new temporary directory. * Add a `BoxStateService` type alias Make it easier to write and read a boxed version of a state service. * Add a helper function to start the state service Make it easier to specify the state service to use an existing state cache directory. * Import `eyre!` macro at the module level Allow it to be used in different places without having to repeat the imports. * Add `load_tip_height_from_state_directory` helper A function to discover the current chain tip height stored in a state cache. * Add helper function to prepare partial sync. state Loads a partially synchronized cached state directory into a temporary directory that can be used by a zebrad instance, and also returns the chain tip block height of that state. * Add `perform_full_sync_starting_from` helper Runs a zebrad with an existing partially synchronized state, and finishes synchronizing it to the network chain tip. * Add function to load transactions from a block Use a provided state service to load all transactions from a block at a specified height. The state service is a generic type parameter, because `zebra_state::service::ReadStateService` is not exported publicly. Using a generic type parameter also allows the service to be wrapped in layers if needed in the future. * Add `load_transactions_from_block_after` helper A function to load transactions from a block stored in a cached state directory. The cached state must be synchronized to a chain tip higher than the requested height. * Add helper function to load some test transactions Given a partially synchronized chain state, it will extend that chain by performing a full synchronization, and obtain some transactions from one of the newly added blocks. * Update `spawn_zebrad_for_rpc_without_initial_peers` Wait until the mempool is activated. * Add method to start lightwalletd with RPC server Returns the lightwalletd instance and the port that it's listening for RPC connections. The instance can reuse an existing cached lightwalletd state if the `LIGHTWALLETD_DATA_DIR` environment variable is set. * Add a `LightwalletdRpcClient` type alias To make it easier to identify the type generated from the Protobuf files. * Add helper function to connect to lightwalletd Prepare an RPC client to send requests to a lightwalletd instance. * Add a `prepare_send_transaction_request` helper Creates a request message for lightwalletd to send a transaction. * Add test to send transactions using lightwalletd Obtain some valid transactions from future blocks and try to send them to a lightwalletd instance connected to a zebrad instance that hasn't seen those transactions yet. The transactions should be successfully queued in Zebra's mempool. * Make `zebra_directory` parameter generic Allow using a `TempDir` or a `PathBuf`. * Move lightwalletd protobuf files Place them closer to the module directory, so that it's clearer that they specify the RPC protocol for lightwalletd, and not Zebra itself. * Don't use coinbase transactions in the test Coinbase transactions are rejected by the mempool. * Don't remove state lock file It is removed automatically by Zebra when it shuts down, so if it exists it should be reported as a bug. * Force mempool to be enabled in Zebrad instance Speed up the initialization of the Zebrad instance used for lightwalletd to connect to. * Refactor to create `LIGHTWALLETD_DATA_DIR_VAR` Document how the environment variable can be used to speed up the test. * Check for process errors in spawned Zebra instance Enable checking for known process failure messages. * Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant Document why it exists and how the choice of the value affects the test. * Add `LIGHTWALLETD_TEST_TIMEOUT` constant And use it for the Zebrad and the Lightwalletd instances used in the send transaction integration test. * Check `lightwalletd` process for errors Enable checking the lightwalletd process for known failure messages. * Update `tonic` and `prost` dependencies Use the latest version and fix CI failures because `rustfmt` isn't installed in the build environment. * Create `send_transaction_test` module Move the send transaction using lightwalletd test and its helper functions into a new module. * Move `LIGHTWALLETD_TEST_TIMEOUT` constant Place it in the parent `lightwalletd` module. * Move gRPC helper functions and types to `rpc` mod. Make them more accessible so that they can be used by other tests. * Create a `cached_state` module Move the test utility functions related to using a cached Zebra state into the module. * Move `perform_full_sync_starting_from` to `sync` Keep to closer to the synchronization utility functions. * Move Zebra cached state path variable constant Place it in the `cached_state` module. * Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set Make it part of the set of tests ignored as a whole if no lightwalletd tests should be executed. * Move `spawn_zebrad_for_rpc_without_initial_peers` Place it in the `launch` sub-module. * Rename `rpc` module into `wallet_grpc` Avoid any potential misunderstandings when the name is seen out of context. * Allow duplicate `heck` dependency At least until `structopt` is updated or `zebra-utils` is updated to use `clap` 3. * Fix a deny.toml typo * fix(build): CMake is required by `prost` crate Co-authored-by: teor <teor@riseup.net> Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 16:06:11 -07:00
let (lightwalletd_failure_messages, lightwalletd_ignore_messages) =
test_type.lightwalletd_failure_messages();
add(test): Integration test to send transactions using lightwalletd (#4068) * Export the `zebra_state::Config::db_path` method Make it easier for tests to discover the sub-directory used to store the chain state data. * Generate code for interfacing with lightwalletd Use the `tonic-build` crate to generate Rust code for communicating with lightwalletd using gRPC. The `*.proto` files were obtained from the Zcash lightwalletd repository. * Use `block::Height` instead of `Height` Import the `block` instead to make it slightly clearer. * Add helper function to remove a file if it exists Try to remove it and ignore an error if it says that the file doesn't exist. This will be used later to remove the lock file from a copied chain state directory. * Add helper function to copy chain state dirs Copy an existing chain state directory into a new temporary directory. * Add a `BoxStateService` type alias Make it easier to write and read a boxed version of a state service. * Add a helper function to start the state service Make it easier to specify the state service to use an existing state cache directory. * Import `eyre!` macro at the module level Allow it to be used in different places without having to repeat the imports. * Add `load_tip_height_from_state_directory` helper A function to discover the current chain tip height stored in a state cache. * Add helper function to prepare partial sync. state Loads a partially synchronized cached state directory into a temporary directory that can be used by a zebrad instance, and also returns the chain tip block height of that state. * Add `perform_full_sync_starting_from` helper Runs a zebrad with an existing partially synchronized state, and finishes synchronizing it to the network chain tip. * Add function to load transactions from a block Use a provided state service to load all transactions from a block at a specified height. The state service is a generic type parameter, because `zebra_state::service::ReadStateService` is not exported publicly. Using a generic type parameter also allows the service to be wrapped in layers if needed in the future. * Add `load_transactions_from_block_after` helper A function to load transactions from a block stored in a cached state directory. The cached state must be synchronized to a chain tip higher than the requested height. * Add helper function to load some test transactions Given a partially synchronized chain state, it will extend that chain by performing a full synchronization, and obtain some transactions from one of the newly added blocks. * Update `spawn_zebrad_for_rpc_without_initial_peers` Wait until the mempool is activated. * Add method to start lightwalletd with RPC server Returns the lightwalletd instance and the port that it's listening for RPC connections. The instance can reuse an existing cached lightwalletd state if the `LIGHTWALLETD_DATA_DIR` environment variable is set. * Add a `LightwalletdRpcClient` type alias To make it easier to identify the type generated from the Protobuf files. * Add helper function to connect to lightwalletd Prepare an RPC client to send requests to a lightwalletd instance. * Add a `prepare_send_transaction_request` helper Creates a request message for lightwalletd to send a transaction. * Add test to send transactions using lightwalletd Obtain some valid transactions from future blocks and try to send them to a lightwalletd instance connected to a zebrad instance that hasn't seen those transactions yet. The transactions should be successfully queued in Zebra's mempool. * Make `zebra_directory` parameter generic Allow using a `TempDir` or a `PathBuf`. * Move lightwalletd protobuf files Place them closer to the module directory, so that it's clearer that they specify the RPC protocol for lightwalletd, and not Zebra itself. * Don't use coinbase transactions in the test Coinbase transactions are rejected by the mempool. * Don't remove state lock file It is removed automatically by Zebra when it shuts down, so if it exists it should be reported as a bug. * Force mempool to be enabled in Zebrad instance Speed up the initialization of the Zebrad instance used for lightwalletd to connect to. * Refactor to create `LIGHTWALLETD_DATA_DIR_VAR` Document how the environment variable can be used to speed up the test. * Check for process errors in spawned Zebra instance Enable checking for known process failure messages. * Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant Document why it exists and how the choice of the value affects the test. * Add `LIGHTWALLETD_TEST_TIMEOUT` constant And use it for the Zebrad and the Lightwalletd instances used in the send transaction integration test. * Check `lightwalletd` process for errors Enable checking the lightwalletd process for known failure messages. * Update `tonic` and `prost` dependencies Use the latest version and fix CI failures because `rustfmt` isn't installed in the build environment. * Create `send_transaction_test` module Move the send transaction using lightwalletd test and its helper functions into a new module. * Move `LIGHTWALLETD_TEST_TIMEOUT` constant Place it in the parent `lightwalletd` module. * Move gRPC helper functions and types to `rpc` mod. Make them more accessible so that they can be used by other tests. * Create a `cached_state` module Move the test utility functions related to using a cached Zebra state into the module. * Move `perform_full_sync_starting_from` to `sync` Keep to closer to the synchronization utility functions. * Move Zebra cached state path variable constant Place it in the `cached_state` module. * Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set Make it part of the set of tests ignored as a whole if no lightwalletd tests should be executed. * Move `spawn_zebrad_for_rpc_without_initial_peers` Place it in the `launch` sub-module. * Rename `rpc` module into `wallet_grpc` Avoid any potential misunderstandings when the name is seen out of context. * Allow duplicate `heck` dependency At least until `structopt` is updated or `zebra-utils` is updated to use `clap` 3. * Fix a deny.toml typo * fix(build): CMake is required by `prost` crate Co-authored-by: teor <teor@riseup.net> Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 16:06:11 -07:00
let mut lightwalletd = lightwalletd_dir
.spawn_lightwalletd_child(lightwalletd_state_path, arguments)?
.with_timeout(test_type.lightwalletd_timeout())
.with_failure_regex_iter(lightwalletd_failure_messages, lightwalletd_ignore_messages);
add(test): Integration test to send transactions using lightwalletd (#4068) * Export the `zebra_state::Config::db_path` method Make it easier for tests to discover the sub-directory used to store the chain state data. * Generate code for interfacing with lightwalletd Use the `tonic-build` crate to generate Rust code for communicating with lightwalletd using gRPC. The `*.proto` files were obtained from the Zcash lightwalletd repository. * Use `block::Height` instead of `Height` Import the `block` instead to make it slightly clearer. * Add helper function to remove a file if it exists Try to remove it and ignore an error if it says that the file doesn't exist. This will be used later to remove the lock file from a copied chain state directory. * Add helper function to copy chain state dirs Copy an existing chain state directory into a new temporary directory. * Add a `BoxStateService` type alias Make it easier to write and read a boxed version of a state service. * Add a helper function to start the state service Make it easier to specify the state service to use an existing state cache directory. * Import `eyre!` macro at the module level Allow it to be used in different places without having to repeat the imports. * Add `load_tip_height_from_state_directory` helper A function to discover the current chain tip height stored in a state cache. * Add helper function to prepare partial sync. state Loads a partially synchronized cached state directory into a temporary directory that can be used by a zebrad instance, and also returns the chain tip block height of that state. * Add `perform_full_sync_starting_from` helper Runs a zebrad with an existing partially synchronized state, and finishes synchronizing it to the network chain tip. * Add function to load transactions from a block Use a provided state service to load all transactions from a block at a specified height. The state service is a generic type parameter, because `zebra_state::service::ReadStateService` is not exported publicly. Using a generic type parameter also allows the service to be wrapped in layers if needed in the future. * Add `load_transactions_from_block_after` helper A function to load transactions from a block stored in a cached state directory. The cached state must be synchronized to a chain tip higher than the requested height. * Add helper function to load some test transactions Given a partially synchronized chain state, it will extend that chain by performing a full synchronization, and obtain some transactions from one of the newly added blocks. * Update `spawn_zebrad_for_rpc_without_initial_peers` Wait until the mempool is activated. * Add method to start lightwalletd with RPC server Returns the lightwalletd instance and the port that it's listening for RPC connections. The instance can reuse an existing cached lightwalletd state if the `LIGHTWALLETD_DATA_DIR` environment variable is set. * Add a `LightwalletdRpcClient` type alias To make it easier to identify the type generated from the Protobuf files. * Add helper function to connect to lightwalletd Prepare an RPC client to send requests to a lightwalletd instance. * Add a `prepare_send_transaction_request` helper Creates a request message for lightwalletd to send a transaction. * Add test to send transactions using lightwalletd Obtain some valid transactions from future blocks and try to send them to a lightwalletd instance connected to a zebrad instance that hasn't seen those transactions yet. The transactions should be successfully queued in Zebra's mempool. * Make `zebra_directory` parameter generic Allow using a `TempDir` or a `PathBuf`. * Move lightwalletd protobuf files Place them closer to the module directory, so that it's clearer that they specify the RPC protocol for lightwalletd, and not Zebra itself. * Don't use coinbase transactions in the test Coinbase transactions are rejected by the mempool. * Don't remove state lock file It is removed automatically by Zebra when it shuts down, so if it exists it should be reported as a bug. * Force mempool to be enabled in Zebrad instance Speed up the initialization of the Zebrad instance used for lightwalletd to connect to. * Refactor to create `LIGHTWALLETD_DATA_DIR_VAR` Document how the environment variable can be used to speed up the test. * Check for process errors in spawned Zebra instance Enable checking for known process failure messages. * Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant Document why it exists and how the choice of the value affects the test. * Add `LIGHTWALLETD_TEST_TIMEOUT` constant And use it for the Zebrad and the Lightwalletd instances used in the send transaction integration test. * Check `lightwalletd` process for errors Enable checking the lightwalletd process for known failure messages. * Update `tonic` and `prost` dependencies Use the latest version and fix CI failures because `rustfmt` isn't installed in the build environment. * Create `send_transaction_test` module Move the send transaction using lightwalletd test and its helper functions into a new module. * Move `LIGHTWALLETD_TEST_TIMEOUT` constant Place it in the parent `lightwalletd` module. * Move gRPC helper functions and types to `rpc` mod. Make them more accessible so that they can be used by other tests. * Create a `cached_state` module Move the test utility functions related to using a cached Zebra state into the module. * Move `perform_full_sync_starting_from` to `sync` Keep to closer to the synchronization utility functions. * Move Zebra cached state path variable constant Place it in the `cached_state` module. * Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set Make it part of the set of tests ignored as a whole if no lightwalletd tests should be executed. * Move `spawn_zebrad_for_rpc_without_initial_peers` Place it in the `launch` sub-module. * Rename `rpc` module into `wallet_grpc` Avoid any potential misunderstandings when the name is seen out of context. * Allow duplicate `heck` dependency At least until `structopt` is updated or `zebra-utils` is updated to use `clap` 3. * Fix a deny.toml typo * fix(build): CMake is required by `prost` crate Co-authored-by: teor <teor@riseup.net> Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 16:06:11 -07:00
lightwalletd.expect_stdout_line_matches("Starting gRPC server")?;
if wait_for_blocks {
feat(ci): add `sending_transactions_using_lightwalletd` test to CI (#4267) * feat(ci): add lightwalletd_*_sync tests to CI * feat(ci): add lightwalletd RPC call test * feat(ci): add send transactions test with lwd to CI * fix(ci): create a variable to run transactions test * refactor(ci): use docker in docker This is a workaround for an issue related to disk partitioning, caused by a GCP service called Konlet, while mounting the cached disks to the VM and then to the container * fix(build): persist docker login credentials * fix(ci): get sync height from docker logs instead of gcp * try: use gha cache for faster building * fix(ci): mount disk in container to make it available in vm * fix(build): do not invalidate cache between images * try(docker): invalidate cache as less as possible * fix(ci): GHA terminal is not a TTY * fix(build): do not ignore entrypoint.sh * fix * fix(ci): mount using root priveleges * fix(ci): use existing disk as cached state * fix(ci): wait for disks to get mounted * force rebuild * fix failed force * fix failed commit * WIP * fix(ci): some tests does not use a cached state * wip * refactor(ci): disk names and job segregation * fix(ci): do not name boot and attached disk the same * fix(ci): attach a disk to full sync, to snapshot the state * fix(ci): use correct disk implementations * fix(ci): use different disk name to allow test concurrency * feat(ci): add lightwalledt send transaction test * cleanup(ci): remove extra tests * fix(ci): allow disk concurrency with tests * fix(ci): add considerations for different tests * fix(reusable): last fixes * feat(ci): use reusable workflow for tests * fix(rw): remove nested worflow * fix(rw): minor fixes * force rebuild * fix(rw): do not use an input as job name * fix(rw): remove variable id * fix(ci): remove explicit conditions and id * fix(ci): docker does not need the variable sign ($) to work * fix(ci): mount typo * fix(ci): if a sync fails, always delete the instance This also reduces the amount of jobs needed. * refactor(ci): make all test depend on the same build * fix(ci): some tests require multiple variables * fix(docker): variable substitution * fix(ci): allow to run multiple commits from a PR at once * fix(docker): lower the NETWORK env var for test names * reduce uneeded diff * imp(keys): use better naming for builds_disks * imp(ci): use input defaults * imp(ci): remove test_name in favor of test_id * fix(ci): better key naming * fix(ci): long disk names breaks GCP naming convention * feat(ci): validate local state version with cached state * fix(ci): add condition to run tests * fix: typo * fix: app_name should not be required * fix: zebra_state_path shouldn't be required * fix: reduce diff * fix(ci): checkout to grep local state version * Update .github/workflows/test.yml Co-authored-by: teor <teor@riseup.net> * revert: merge all tests into a single workflow * Remove unused STATE_VERSION env var * fix: minor fixes * fix(ci): make test.patch the same as test * fix(ci): negate the input value * imp(ci): better cached state conditional handling * imp(ci): exit code is captured by `docker run` * fix(deploy): mount disks with better write performance * fix(ci): change sync id to a broader id name * fix(ci): use correct input validation * fix(ci): do not make test with cached state dependant on other * imp(ci): organiza keys better * fix(ci): use appropiate naming * fix(ci): create docker volume before mounting * fix(lint): do not fail on all new changes * imp(ci): do not report in pr review * fix(ci): partition clean disks * fix: typo * fix: test called the wrong way * fix(build): stop using gha cache * ref(ci): validate run condition before calling reusable workflow * fix(ci): use a better filesystem dir and fix other values * fix: linting errors * fix(ci): typo * Revert "fix(build): stop using gha cache" This reverts commit a8fbc5f416df561e58b388e065d1dc9696983508. Cache expiration is a lesser evil than not using caching at all and then failing with a 401 * imp(ci): do not set a default for needs_zebra_state * Update .github/workflows/test.yml Co-authored-by: teor <teor@riseup.net> * fix(deps): remove dependencies * force build * Update .github/workflows/test.yml Co-authored-by: teor <teor@riseup.net> * fix(docker): add RUST_LOG as an ARG and ENV * fix(test): add `#[ignore]` to send transactions test This test needs state then it should be marked as #[ignore] * fix(ci): differentiate between root cache path and its dir * Remove extra `state` directory That was a workaround for an issue that has been fixed. * imp(docs): use better test descriptions Co-authored-by: teor <teor@riseup.net> * fix: reduce unwanted diff with main * fix(ci): make lwd conditions consistent * Remove another extra `state` directory Was also part of a workaround for an issue that has been fixed. * fix(ci): use better conditionals to run test jobs Co-authored-by: teor <teor@riseup.net> * Tweak to support different lightwalletd versions Some versions print `Waiting for block`, and some versions print `Ingestor waiting for block`. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: teor <teor@riseup.net> Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2022-05-05 22:30:38 -07:00
lightwalletd.expect_stdout_line_matches("[Ww]aiting for block")?;
}
add(test): Integration test to send transactions using lightwalletd (#4068) * Export the `zebra_state::Config::db_path` method Make it easier for tests to discover the sub-directory used to store the chain state data. * Generate code for interfacing with lightwalletd Use the `tonic-build` crate to generate Rust code for communicating with lightwalletd using gRPC. The `*.proto` files were obtained from the Zcash lightwalletd repository. * Use `block::Height` instead of `Height` Import the `block` instead to make it slightly clearer. * Add helper function to remove a file if it exists Try to remove it and ignore an error if it says that the file doesn't exist. This will be used later to remove the lock file from a copied chain state directory. * Add helper function to copy chain state dirs Copy an existing chain state directory into a new temporary directory. * Add a `BoxStateService` type alias Make it easier to write and read a boxed version of a state service. * Add a helper function to start the state service Make it easier to specify the state service to use an existing state cache directory. * Import `eyre!` macro at the module level Allow it to be used in different places without having to repeat the imports. * Add `load_tip_height_from_state_directory` helper A function to discover the current chain tip height stored in a state cache. * Add helper function to prepare partial sync. state Loads a partially synchronized cached state directory into a temporary directory that can be used by a zebrad instance, and also returns the chain tip block height of that state. * Add `perform_full_sync_starting_from` helper Runs a zebrad with an existing partially synchronized state, and finishes synchronizing it to the network chain tip. * Add function to load transactions from a block Use a provided state service to load all transactions from a block at a specified height. The state service is a generic type parameter, because `zebra_state::service::ReadStateService` is not exported publicly. Using a generic type parameter also allows the service to be wrapped in layers if needed in the future. * Add `load_transactions_from_block_after` helper A function to load transactions from a block stored in a cached state directory. The cached state must be synchronized to a chain tip higher than the requested height. * Add helper function to load some test transactions Given a partially synchronized chain state, it will extend that chain by performing a full synchronization, and obtain some transactions from one of the newly added blocks. * Update `spawn_zebrad_for_rpc_without_initial_peers` Wait until the mempool is activated. * Add method to start lightwalletd with RPC server Returns the lightwalletd instance and the port that it's listening for RPC connections. The instance can reuse an existing cached lightwalletd state if the `LIGHTWALLETD_DATA_DIR` environment variable is set. * Add a `LightwalletdRpcClient` type alias To make it easier to identify the type generated from the Protobuf files. * Add helper function to connect to lightwalletd Prepare an RPC client to send requests to a lightwalletd instance. * Add a `prepare_send_transaction_request` helper Creates a request message for lightwalletd to send a transaction. * Add test to send transactions using lightwalletd Obtain some valid transactions from future blocks and try to send them to a lightwalletd instance connected to a zebrad instance that hasn't seen those transactions yet. The transactions should be successfully queued in Zebra's mempool. * Make `zebra_directory` parameter generic Allow using a `TempDir` or a `PathBuf`. * Move lightwalletd protobuf files Place them closer to the module directory, so that it's clearer that they specify the RPC protocol for lightwalletd, and not Zebra itself. * Don't use coinbase transactions in the test Coinbase transactions are rejected by the mempool. * Don't remove state lock file It is removed automatically by Zebra when it shuts down, so if it exists it should be reported as a bug. * Force mempool to be enabled in Zebrad instance Speed up the initialization of the Zebrad instance used for lightwalletd to connect to. * Refactor to create `LIGHTWALLETD_DATA_DIR_VAR` Document how the environment variable can be used to speed up the test. * Check for process errors in spawned Zebra instance Enable checking for known process failure messages. * Add `FINISH_PARTIAL_SYNC_TIMEOUT` constant Document why it exists and how the choice of the value affects the test. * Add `LIGHTWALLETD_TEST_TIMEOUT` constant And use it for the Zebrad and the Lightwalletd instances used in the send transaction integration test. * Check `lightwalletd` process for errors Enable checking the lightwalletd process for known failure messages. * Update `tonic` and `prost` dependencies Use the latest version and fix CI failures because `rustfmt` isn't installed in the build environment. * Create `send_transaction_test` module Move the send transaction using lightwalletd test and its helper functions into a new module. * Move `LIGHTWALLETD_TEST_TIMEOUT` constant Place it in the parent `lightwalletd` module. * Move gRPC helper functions and types to `rpc` mod. Make them more accessible so that they can be used by other tests. * Create a `cached_state` module Move the test utility functions related to using a cached Zebra state into the module. * Move `perform_full_sync_starting_from` to `sync` Keep to closer to the synchronization utility functions. * Move Zebra cached state path variable constant Place it in the `cached_state` module. * Skip test if `ZEBRA_TEST_LIGHTWALLETD` is not set Make it part of the set of tests ignored as a whole if no lightwalletd tests should be executed. * Move `spawn_zebrad_for_rpc_without_initial_peers` Place it in the `launch` sub-module. * Rename `rpc` module into `wallet_grpc` Avoid any potential misunderstandings when the name is seen out of context. * Allow duplicate `heck` dependency At least until `structopt` is updated or `zebra-utils` is updated to use `clap` 3. * Fix a deny.toml typo * fix(build): CMake is required by `prost` crate Co-authored-by: teor <teor@riseup.net> Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
2022-04-27 16:06:11 -07:00
Ok((lightwalletd, lightwalletd_rpc_port))
}
/// Connect to a lightwalletd RPC instance.
pub async fn connect_to_lightwalletd(lightwalletd_rpc_port: u16) -> Result<LightwalletdRpcClient> {
let lightwalletd_rpc_address = format!("http://127.0.0.1:{lightwalletd_rpc_port}");
let rpc_client = LightwalletdRpcClient::connect(lightwalletd_rpc_address).await?;
Ok(rpc_client)
}