zebra/zebrad/src/commands/start.rs

557 lines
22 KiB
Rust
Raw Normal View History

//! `start` subcommand - entry point for starting a zebra node
//!
//! ## Application Structure
//!
feat(net): Cache a list of useful peers on disk (#6739) * Rewrite some state cache docs to clarify * Add a zebra_network::Config.cache_dir for peer address caches * Add new config test files and fix config test failure message * Create some zebra-chain and zebra-network convenience functions * Add methods for reading and writing the peer address cache * Add cached disk peers to the initial peers list * Add metrics and logging for loading and storing the peer cache * Replace log of useless redacted peer IP addresses * Limit the peer cache minimum and maximum size, don't write empty caches * Add a cacheable_peers() method to the address book * Add a peer disk cache updater task to the peer set tasks * Document that the peer cache is shared by multiple instances unless configured otherwise * Disable peer cache read/write in disconnected tests * Make initial peer cache updater sleep shorter for tests * Add unit tests for reading and writing the peer cache * Update the task list in the start command docs * Modify the existing persistent acceptance test to check for peer caches * Update the peer cache directory when writing test configs * Add a CacheDir type so the default config can be enabled, but tests can disable it * Update tests to use the CacheDir config type * Rename some CacheDir internals * Add config file test cases for each kind of CacheDir config * Panic if the config contains invalid socket addresses, rather than continuing * Add a network directory to state cache directory contents tests * Add new network.cache_dir config to the config parsing tests
2023-06-06 01:28:14 -07:00
//! A zebra node consists of the following major services and tasks:
//!
//! Peers:
//! * Peer Connection Pool Service
//! * primary external interface for outbound requests from this node to remote peers
//! * accepts requests from services and tasks in this node, and sends them to remote peers
//! * Peer Discovery Service
//! * maintains a list of peer addresses, and connection priority metadata
//! * discovers new peer addresses from existing peer connections
//! * initiates new outbound peer connections in response to demand from tasks within this node
feat(net): Cache a list of useful peers on disk (#6739) * Rewrite some state cache docs to clarify * Add a zebra_network::Config.cache_dir for peer address caches * Add new config test files and fix config test failure message * Create some zebra-chain and zebra-network convenience functions * Add methods for reading and writing the peer address cache * Add cached disk peers to the initial peers list * Add metrics and logging for loading and storing the peer cache * Replace log of useless redacted peer IP addresses * Limit the peer cache minimum and maximum size, don't write empty caches * Add a cacheable_peers() method to the address book * Add a peer disk cache updater task to the peer set tasks * Document that the peer cache is shared by multiple instances unless configured otherwise * Disable peer cache read/write in disconnected tests * Make initial peer cache updater sleep shorter for tests * Add unit tests for reading and writing the peer cache * Update the task list in the start command docs * Modify the existing persistent acceptance test to check for peer caches * Update the peer cache directory when writing test configs * Add a CacheDir type so the default config can be enabled, but tests can disable it * Update tests to use the CacheDir config type * Rename some CacheDir internals * Add config file test cases for each kind of CacheDir config * Panic if the config contains invalid socket addresses, rather than continuing * Add a network directory to state cache directory contents tests * Add new network.cache_dir config to the config parsing tests
2023-06-06 01:28:14 -07:00
//! * Peer Cache Service
//! * Reads previous peer cache on startup, and adds it to the configured DNS seed peers
//! * Periodically updates the peer cache on disk from the latest address book state
//!
//! Blocks & Mempool Transactions:
//! * Consensus Service
//! * handles all validation logic for the node
//! * verifies blocks using zebra-chain, then stores verified blocks in zebra-state
//! * verifies mempool and block transactions using zebra-chain and zebra-script,
//! and returns verified mempool transactions for mempool storage
//! * Inbound Service
//! * primary external interface for inbound peer requests to this node
//! * handles requests from peers for network data, chain data, and mempool transactions
//! * spawns download and verify tasks for each gossiped block
//! * sends gossiped transactions to the mempool service
//!
//! Blocks:
//! * Sync Task
//! * runs in the background and continuously queries the network for
//! new blocks to be verified and added to the local state
//! * spawns download and verify tasks for each crawled block
//! * State Service
//! * contextually verifies blocks
//! * handles in-memory storage of multiple non-finalized chains
//! * handles permanent storage of the best finalized chain
//! * Old State Version Cleanup Task
//! * deletes outdated state versions
//! * Block Gossip Task
//! * runs in the background and continuously queries the state for
//! newly committed blocks to be gossiped to peers
//! * Progress Task
//! * logs progress towards the chain tip
//!
feat(mine): Add an internal Zcash miner to Zebra (#8136) * Patch equihash to use the solver branch * Add an internal-miner feature and set up its dependencies * Remove 'Experimental' from mining RPC docs * Fix a nightly clippy::question_mark lint * Move a byte array utility function to zebra-chain * fixup! Add an internal-miner feature and set up its dependencies * Add an equihash::Solution::solve() method with difficulty checks * Check solution is valid before returning it * Add a TODO to check for peers before mining * Move config validation into GetBlockTemplateRpcImpl::new() * fixup! fixup! Add an internal-miner feature and set up its dependencies * Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls * Start adding an internal miner component * Add the miner task to the start command * Add basic miner code * Split out a method to mine one block * Spawn to a blocking thread * Wait until a valid template is available * Handle shutdown * Run mining on low priority threads * Ignore some invalid solutions * Use a difference nonce for each solver thread * Update TODOs * Change the patch into a renamed dependency to simplify crate releases * Clean up instrumentation and TODOs * Make RPC instances cloneable and clean up generics * Make LongPollId Copy so it's easier to use * Add API to restart mining if there's a new block template * Actually restart mining if there's a new block template * Tidy instrumentation * fixup! Move config validation into GetBlockTemplateRpcImpl::new() * fixup! Make RPC instances cloneable and clean up generics * Run the template generator and one miner concurrently * Reduce logging * Fix a bug in getblocktemplate RPC tip change detection * Work around some watch channel change bugs * Rate-limit template changes in the receiver * Run one mining solver per available core * Use updated C code with double-free protection * Update to the latest solver branch * Return and submit all valid solutions * Document what INPUT_LENGTH means * Fix watch channel change detection * Don't return early when a mining task fails * Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness * Make existing parallelism docs and configs consistent * Add a mining parallelism config * Use the minimum of the configured or available threads for mining * Ignore optional feature fields in tests * Downgrade some frequent logs to debug * Document new zebrad features and tasks * Describe the internal-miner feature in the CHANGELOG * Update dependency to de-duplicate equihash solutions * Use futures::StreamExt instead of TryStreamExt * Fix a panic message typo
2024-01-11 06:41:01 -08:00
//! Shielded Scanning:
//! * Shielded Scanner Task
//! * if the user has configured Zebra with their shielded viewing keys, scans new and existing
//! blocks for transactions that use those keys
//!
//! Block Mining:
//! * Internal Miner Task
//! * if the user has configured Zebra to mine blocks, spawns tasks to generate new blocks,
//! and submits them for verification. This automatically shares these new blocks with peers.
//!
//! Mempool Transactions:
//! * Mempool Service
//! * activates when the syncer is near the chain tip
//! * spawns download and verify tasks for each crawled or gossiped transaction
//! * handles in-memory storage of unmined transactions
//! * Queue Checker Task
//! * runs in the background, polling the mempool to store newly verified transactions
//! * Transaction Gossip Task
//! * runs in the background and gossips newly added mempool transactions
//! to peers
//!
//! Remote Procedure Calls:
//! * JSON-RPC Service
//! * answers RPC client requests using the State Service and Mempool Service
//! * submits client transactions to the node's mempool
//!
feat(ui): Add a terminal-based progress bar to Zebra (#6235) * Implement Display and to_string() for NetworkUpgrade * Add a progress-bar feature to zebrad * Add the progress bar writer to the tracing component * Add a block progress bar transmitter * Correctly shut down the progress bar, and shut it down on an interrupt * Make it clearer that the progress task never exits * Add a config for writing logs to a file * Add a progress-bar feature to zebra-network * Add a progress bar for the address book size * Add progress bars for never attempted and failed peers * Add an optional limit and label to connection counters * Add open connection progress bars * Improve CheckpointList API and CheckpointVerifier debugging * Add checkpoint index and checkpoint queue progress bars * Security: Limit the number of non-finalized chains tracked by Zebra * Make some NonFinalizedState methods available with proptest-impl * Add a non-finalized chain count progress bar * Track the last fork height for newly forked chains * Add a should_count_metrics to Chain * Add a display method for PartialCumulativeWork * Add a progress bar for each chain fork * Add a NonFinalizedState::disable_metrics() method and switch to using it * Move metrics out of Chain because we can't update Arc<Chain> * Fix: consistently use best chain order when searching chains * Track Chain progress bars in NonFinalizedState * Display work as bits, not a multiple of the target difficulty * Handle negative fork lengths by reporting "No fork" * Correctly disable unused fork bars * clippy: rewrite using `match _.cmp(_) { ... }` * Initial mempool progress bar implementation * Update Cargo.lock * Add the actual transaction size as a description to the cost bar * Only show mempool progress bars after first activation * Add queued and rejected mempool progress bars * Clarify cost note is actual size * Add tracing.log_file config and progress-bar feature to zebrad docs * Derive Clone for Chain * Upgrade to howudoin 0.1.2 and remove some bug workarounds * Directly call the debug formatter to Display a Network Co-authored-by: Arya <aryasolhi@gmail.com> * Rename the address count metric to num_addresses Co-authored-by: Arya <aryasolhi@gmail.com> * Simplify reverse checkpoint lookup Co-authored-by: Arya <aryasolhi@gmail.com> * Simplify progress bar shutdown code Co-authored-by: Arya <aryasolhi@gmail.com> * Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE * Document that the progress task runs forever * Fix progress log formatting * If progress-bar is on, log to a file by default * Create missing directories for log files * Add file security docs for running Zebra with elevated permissions * Document automatic log file, spell progress-bar correctly --------- Co-authored-by: Arya <aryasolhi@gmail.com>
2023-04-13 01:42:17 -07:00
//! Zebra also has diagnostic support:
//! * [metrics](https://github.com/ZcashFoundation/zebra/blob/main/book/src/user/metrics.md)
//! * [tracing](https://github.com/ZcashFoundation/zebra/blob/main/book/src/user/tracing.md)
feat(ui): Add a terminal-based progress bar to Zebra (#6235) * Implement Display and to_string() for NetworkUpgrade * Add a progress-bar feature to zebrad * Add the progress bar writer to the tracing component * Add a block progress bar transmitter * Correctly shut down the progress bar, and shut it down on an interrupt * Make it clearer that the progress task never exits * Add a config for writing logs to a file * Add a progress-bar feature to zebra-network * Add a progress bar for the address book size * Add progress bars for never attempted and failed peers * Add an optional limit and label to connection counters * Add open connection progress bars * Improve CheckpointList API and CheckpointVerifier debugging * Add checkpoint index and checkpoint queue progress bars * Security: Limit the number of non-finalized chains tracked by Zebra * Make some NonFinalizedState methods available with proptest-impl * Add a non-finalized chain count progress bar * Track the last fork height for newly forked chains * Add a should_count_metrics to Chain * Add a display method for PartialCumulativeWork * Add a progress bar for each chain fork * Add a NonFinalizedState::disable_metrics() method and switch to using it * Move metrics out of Chain because we can't update Arc<Chain> * Fix: consistently use best chain order when searching chains * Track Chain progress bars in NonFinalizedState * Display work as bits, not a multiple of the target difficulty * Handle negative fork lengths by reporting "No fork" * Correctly disable unused fork bars * clippy: rewrite using `match _.cmp(_) { ... }` * Initial mempool progress bar implementation * Update Cargo.lock * Add the actual transaction size as a description to the cost bar * Only show mempool progress bars after first activation * Add queued and rejected mempool progress bars * Clarify cost note is actual size * Add tracing.log_file config and progress-bar feature to zebrad docs * Derive Clone for Chain * Upgrade to howudoin 0.1.2 and remove some bug workarounds * Directly call the debug formatter to Display a Network Co-authored-by: Arya <aryasolhi@gmail.com> * Rename the address count metric to num_addresses Co-authored-by: Arya <aryasolhi@gmail.com> * Simplify reverse checkpoint lookup Co-authored-by: Arya <aryasolhi@gmail.com> * Simplify progress bar shutdown code Co-authored-by: Arya <aryasolhi@gmail.com> * Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE * Document that the progress task runs forever * Fix progress log formatting * If progress-bar is on, log to a file by default * Create missing directories for log files * Add file security docs for running Zebra with elevated permissions * Document automatic log file, spell progress-bar correctly --------- Co-authored-by: Arya <aryasolhi@gmail.com>
2023-04-13 01:42:17 -07:00
//! * [progress-bar](https://docs.rs/howudoin/0.1.1/howudoin)
//!
//! Some of the diagnostic features are optional, and need to be enabled at compile-time.
fix(zebrad): accept default subcommand arguments and print consistent usage information for top-level 'help' subcommand (#6801) * updates Cargo.toml * Migrate to abscissa 0.7.0 * Avoid panic from calling color_eyre::install twice * Uses 'start' as the default subcommand * updates default cmd logic * Fixes minor cli issues * removes outdated check in acceptance test * Adds a test for process_cli_args, fixes version_args test. Adds -V to process_cli_args match case * Revert "fix(clippy): Silence future-incompat warnings until we upgrade Abscissa (#6024)" This reverts commit dd90f79b4824af6b1ae22ee69abdf0f605da84cd. * Drops the worker guard to flush logs when zebra shuts down * Adds cargo feature to clap * restores process_cli_args * updates deny.toml * Updates EntryPoint help template * Updates subcommand help msgs * removes trailing whitespace, capitalizes sentences * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * revert parts of revert "Revert fix(clippy): Silence future-incompat warnings until we upgrade Abscissa" * Applies suggestions from code review * Moves EntryPoint to its own module * fixes version_args test * Updates changelog * Prunes redundant test cases * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Revert "Prunes redundant test cases" This reverts commit 3f7397918489144805c17d0594775aa699e87b9d. * Update zebrad/src/commands/entry_point.rs Co-authored-by: teor <teor@riseup.net> * Add missing import * Updates `process_cli_args` to return a result --------- Co-authored-by: teor <teor@riseup.net>
2023-06-06 23:03:42 -07:00
use abscissa_core::{config, Command, FrameworkError, Runnable};
use color_eyre::eyre::{eyre, Report};
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
use futures::FutureExt;
use tokio::{pin, select, sync::oneshot};
use tower::{builder::ServiceBuilder, util::BoxService};
use tracing_futures::Instrument;
use zebra_consensus::router::BackgroundTaskHandles;
feature(rpc): add an rpc server to Zebra (#3589) * feature(rpc): add an rpc component * feat(rpc): add a stub for getblockchaininfo This is the first RPC used by lightwalletd, so we need it for testing. * fix(rpc): remove non-standard "jsonrpc: 1.0" from lightwalletd * fix(rpc): re-enable default RPC security checks * deps(rpc): remove not needed dependency * fix(rpc): check if RPC task has stopped * fix(rpc): reduce config by using Option * fix(rpc): use tokio executor * security(rpc): turn off rpc by default * docs(rpc): update a TODO comment Co-authored-by: teor <teor@riseup.net> * fix(rpc): blocking tasks Co-authored-by: teor <teor@riseup.net> * rename(rpc): rpc.rs to methods.rs * refactor(rpc): move the server to the zebra-rpc crate * fix(rpc): clippy derive Default for RPC Config * fix(dependencies): remove unused dependency features in zebra-rpc We expect to use all the listed tokio features to implement and test RPC methods. * doc(rpc): fix testnet port, add security note * fix(rpc): change Rust function names and update method doc TODOs * fix(rpc): add "TODO" to fake RPC responses * doc(rpc): update module docs * fix(rpc): simplify server struct derives * fix(rpc): simplify server code * doc(rpc): explain how request fixes securely handle user-supplied data * refactor(rpc): move the compatibility fix to a separate module * fix(rpc): move the open log inside the spawn, and instrument it * doc(rpc): fix toml format and provide a config example Co-authored-by: teor <teor@riseup.net>
2022-02-22 03:26:29 -08:00
use zebra_rpc::server::RpcServer;
use crate::{
application::{build_version, user_agent},
components::{
inbound::{self, InboundSetupData, MAX_INBOUND_RESPONSE_TIME},
mempool::{self, Mempool},
change(state): Write non-finalized blocks to the state in a separate thread, to avoid network and RPC hangs (#5257) * Add a new block commit task and channels, that don't do anything yet * Add last_block_hash_sent to the state service, to avoid database accesses * Update last_block_hash_sent regardless of commit errors * Rename a field to StateService.max_queued_finalized_height * Commit finalized blocks to the state in a separate task * Check for panics in the block write task * Wait for the block commit task in tests, and check for errors * Always run a proptest that sleeps once * Add extra debugging to state shutdowns * Work around a RocksDB shutdown bug * Close the finalized block channel when we're finished with it * Only reset state queue once per error * Update some TODOs * Add a module doc comment * Drop channels and check for closed channels in the block commit task * Close state channels and tasks on drop * Remove some duplicate fields across StateService and ReadStateService * Try tweaking the shutdown steps * Update and clarify some comments * Clarify another comment * Don't try to cancel RocksDB background work on drop * Fix up some comments * Remove some duplicate code * Remove redundant workarounds for shutdown issues * Remode a redundant channel close in the block commit task * Remove a mistaken `!force` shutdown condition * Remove duplicate force-shutdown code and explain it better * Improve RPC error logging * Wait for chain tip updates in the RPC tests * Wait 2 seconds for chain tip updates before skipping them * Remove an unnecessary block_in_place() * Fix some test error messages that were changed by earlier fixes * Expand some comments, fix typos Co-authored-by: Marek <mail@marek.onl> * Actually drop children of failed blocks * Explain why we drop descendants of failed blocks * Clarify a comment * Wait for chain tip updates in a failing test on macOS * Clean duplicate finalized blocks when the non-finalized state activates * Send an error when receiving a duplicate finalized block * Update checkpoint block behaviour, document its consensus rule * Wait for chain tip changes in inbound_block_height_lookahead_limit test * Wait for the genesis block to commit in the fake peer set mempool tests * Disable unreliable mempool verification check in the send transaction test * Appease rustfmt * Use clear_finalized_block_queue() everywhere that blocks are dropped * Document how Finalized and NonFinalized clones are different * sends non-finalized blocks to the block write task * passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState * Update zebra-state/src/service/write.rs Co-authored-by: teor <teor@riseup.net> * updates comments, renames send_process_queued, other minor cleanup * update assert_block_can_be_validated comment * removes `mem` field from StateService * removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state * updates tests that use the disk to use read_service.db instead * moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service * changes `contextual_validity` to get the network from the finalized_state instead of another param * swaps out StateService with FinalizedState and NonFinalizedState in tests * adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain * removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send * makes parent_errors_map an indexmap * clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped * sends non-finalized blocks to the block write task * passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState * updates comments, renames send_process_queued, other minor cleanup * Update zebra-state/src/service/write.rs Co-authored-by: teor <teor@riseup.net> * update assert_block_can_be_validated comment * removes `mem` field from StateService * removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state * updates tests that use the disk to use read_service.db instead * moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service * changes `contextual_validity` to get the network from the finalized_state instead of another param * swaps out StateService with FinalizedState and NonFinalizedState in tests * adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain * removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send * makes parent_errors_map an indexmap * clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped * removes duplicate field definitions on StateService that were a result of a bad merge * update NotReadyToBeCommitted error message * Appear rustfmt * Fix doc links * Rename a function to initial_contextual_validity() * Do error tasks on Err, and success tasks on Ok * Simplify parent_error_map truncation * Rewrite best_tip() to use tip() * Rename latest_mem() to latest_non_finalized_state() ```sh fastmod latest_mem latest_non_finalized_state zebra* cargo fmt --all ``` * Simplify latest_non_finalized_state() using a new WatchReceiver API * Expand some error messages * Send the result after updating the channels, and document why * wait for chain_tip_update before cancelling download in mempool_cancel_mined * adds `sent_non_finalized_block_hashes` field to StateService * adds batched sent_hash insertions and checks sent hashes in queue_and_commit_non_finalized before adding a block to the queue * check that the `curr_buf` in SentHashes is not empty before pushing it to the `sent_bufs` * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Fix rustfmt * Check for finalized block heights using zs_contains() * adds known_utxos field to SentHashes * updates comment on SentHashes.add method * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * return early when there's a duplicate hash in QueuedBlocks.queue instead of panicking * Make finalized UTXOs near the final checkpoint available for full block verification * Replace a checkpoint height literal with the actual config * Update mainnet and testnet checkpoints - 7 October 2022 * Fix some state service init arguments * Allow more lookahead in the downloader, but less lookahead in the syncer * Add the latest config to the tests, and fix the latest config check * Increase the number of finalized blocks checked for non-finalized block UTXO spends * fix(log): reduce verbose logs for block commits (#5348) * Remove some verbose block write channel logs * Only warn about tracing endpoint if the address is actually set * Use CloneError instead of formatting a non-cloneable error Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Increase block verify timeout * Work around a known block timeout bug by using a shorter timeout Co-authored-by: teor <teor@riseup.net> Co-authored-by: Marek <mail@marek.onl> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-10-11 12:25:45 -07:00
sync::{self, show_block_chain_progress, VERIFICATION_PIPELINE_SCALING_MULTIPLIER},
tokio::{RuntimeRun, TokioComponent},
ChainSync, Inbound,
},
config::ZebradConfig,
prelude::*,
};
fix(zebrad): accept default subcommand arguments and print consistent usage information for top-level 'help' subcommand (#6801) * updates Cargo.toml * Migrate to abscissa 0.7.0 * Avoid panic from calling color_eyre::install twice * Uses 'start' as the default subcommand * updates default cmd logic * Fixes minor cli issues * removes outdated check in acceptance test * Adds a test for process_cli_args, fixes version_args test. Adds -V to process_cli_args match case * Revert "fix(clippy): Silence future-incompat warnings until we upgrade Abscissa (#6024)" This reverts commit dd90f79b4824af6b1ae22ee69abdf0f605da84cd. * Drops the worker guard to flush logs when zebra shuts down * Adds cargo feature to clap * restores process_cli_args * updates deny.toml * Updates EntryPoint help template * Updates subcommand help msgs * removes trailing whitespace, capitalizes sentences * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * revert parts of revert "Revert fix(clippy): Silence future-incompat warnings until we upgrade Abscissa" * Applies suggestions from code review * Moves EntryPoint to its own module * fixes version_args test * Updates changelog * Prunes redundant test cases * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Revert "Prunes redundant test cases" This reverts commit 3f7397918489144805c17d0594775aa699e87b9d. * Update zebrad/src/commands/entry_point.rs Co-authored-by: teor <teor@riseup.net> * Add missing import * Updates `process_cli_args` to return a result --------- Co-authored-by: teor <teor@riseup.net>
2023-06-06 23:03:42 -07:00
/// Start the application (default command)
#[derive(Command, Debug, Default, clap::Parser)]
2020-06-16 11:02:01 -07:00
pub struct StartCmd {
/// Filter strings which override the config file and defaults
fix(zebrad): accept default subcommand arguments and print consistent usage information for top-level 'help' subcommand (#6801) * updates Cargo.toml * Migrate to abscissa 0.7.0 * Avoid panic from calling color_eyre::install twice * Uses 'start' as the default subcommand * updates default cmd logic * Fixes minor cli issues * removes outdated check in acceptance test * Adds a test for process_cli_args, fixes version_args test. Adds -V to process_cli_args match case * Revert "fix(clippy): Silence future-incompat warnings until we upgrade Abscissa (#6024)" This reverts commit dd90f79b4824af6b1ae22ee69abdf0f605da84cd. * Drops the worker guard to flush logs when zebra shuts down * Adds cargo feature to clap * restores process_cli_args * updates deny.toml * Updates EntryPoint help template * Updates subcommand help msgs * removes trailing whitespace, capitalizes sentences * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * revert parts of revert "Revert fix(clippy): Silence future-incompat warnings until we upgrade Abscissa" * Applies suggestions from code review * Moves EntryPoint to its own module * fixes version_args test * Updates changelog * Prunes redundant test cases * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Revert "Prunes redundant test cases" This reverts commit 3f7397918489144805c17d0594775aa699e87b9d. * Update zebrad/src/commands/entry_point.rs Co-authored-by: teor <teor@riseup.net> * Add missing import * Updates `process_cli_args` to return a result --------- Co-authored-by: teor <teor@riseup.net>
2023-06-06 23:03:42 -07:00
#[clap(help = "tracing filters which override the zebrad.toml config")]
filters: Vec<String>,
}
2020-06-16 11:02:01 -07:00
impl StartCmd {
2020-06-15 17:07:55 -07:00
async fn start(&self) -> Result<(), Report> {
fix(zebrad): accept default subcommand arguments and print consistent usage information for top-level 'help' subcommand (#6801) * updates Cargo.toml * Migrate to abscissa 0.7.0 * Avoid panic from calling color_eyre::install twice * Uses 'start' as the default subcommand * updates default cmd logic * Fixes minor cli issues * removes outdated check in acceptance test * Adds a test for process_cli_args, fixes version_args test. Adds -V to process_cli_args match case * Revert "fix(clippy): Silence future-incompat warnings until we upgrade Abscissa (#6024)" This reverts commit dd90f79b4824af6b1ae22ee69abdf0f605da84cd. * Drops the worker guard to flush logs when zebra shuts down * Adds cargo feature to clap * restores process_cli_args * updates deny.toml * Updates EntryPoint help template * Updates subcommand help msgs * removes trailing whitespace, capitalizes sentences * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * revert parts of revert "Revert fix(clippy): Silence future-incompat warnings until we upgrade Abscissa" * Applies suggestions from code review * Moves EntryPoint to its own module * fixes version_args test * Updates changelog * Prunes redundant test cases * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Revert "Prunes redundant test cases" This reverts commit 3f7397918489144805c17d0594775aa699e87b9d. * Update zebrad/src/commands/entry_point.rs Co-authored-by: teor <teor@riseup.net> * Add missing import * Updates `process_cli_args` to return a result --------- Co-authored-by: teor <teor@riseup.net>
2023-06-06 23:03:42 -07:00
let config = APPLICATION.config();
2020-06-15 17:07:55 -07:00
info!("initializing node state");
let (_, max_checkpoint_height) = zebra_consensus::router::init_checkpoint_list(
change(state): Write non-finalized blocks to the state in a separate thread, to avoid network and RPC hangs (#5257) * Add a new block commit task and channels, that don't do anything yet * Add last_block_hash_sent to the state service, to avoid database accesses * Update last_block_hash_sent regardless of commit errors * Rename a field to StateService.max_queued_finalized_height * Commit finalized blocks to the state in a separate task * Check for panics in the block write task * Wait for the block commit task in tests, and check for errors * Always run a proptest that sleeps once * Add extra debugging to state shutdowns * Work around a RocksDB shutdown bug * Close the finalized block channel when we're finished with it * Only reset state queue once per error * Update some TODOs * Add a module doc comment * Drop channels and check for closed channels in the block commit task * Close state channels and tasks on drop * Remove some duplicate fields across StateService and ReadStateService * Try tweaking the shutdown steps * Update and clarify some comments * Clarify another comment * Don't try to cancel RocksDB background work on drop * Fix up some comments * Remove some duplicate code * Remove redundant workarounds for shutdown issues * Remode a redundant channel close in the block commit task * Remove a mistaken `!force` shutdown condition * Remove duplicate force-shutdown code and explain it better * Improve RPC error logging * Wait for chain tip updates in the RPC tests * Wait 2 seconds for chain tip updates before skipping them * Remove an unnecessary block_in_place() * Fix some test error messages that were changed by earlier fixes * Expand some comments, fix typos Co-authored-by: Marek <mail@marek.onl> * Actually drop children of failed blocks * Explain why we drop descendants of failed blocks * Clarify a comment * Wait for chain tip updates in a failing test on macOS * Clean duplicate finalized blocks when the non-finalized state activates * Send an error when receiving a duplicate finalized block * Update checkpoint block behaviour, document its consensus rule * Wait for chain tip changes in inbound_block_height_lookahead_limit test * Wait for the genesis block to commit in the fake peer set mempool tests * Disable unreliable mempool verification check in the send transaction test * Appease rustfmt * Use clear_finalized_block_queue() everywhere that blocks are dropped * Document how Finalized and NonFinalized clones are different * sends non-finalized blocks to the block write task * passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState * Update zebra-state/src/service/write.rs Co-authored-by: teor <teor@riseup.net> * updates comments, renames send_process_queued, other minor cleanup * update assert_block_can_be_validated comment * removes `mem` field from StateService * removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state * updates tests that use the disk to use read_service.db instead * moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service * changes `contextual_validity` to get the network from the finalized_state instead of another param * swaps out StateService with FinalizedState and NonFinalizedState in tests * adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain * removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send * makes parent_errors_map an indexmap * clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped * sends non-finalized blocks to the block write task * passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState * updates comments, renames send_process_queued, other minor cleanup * Update zebra-state/src/service/write.rs Co-authored-by: teor <teor@riseup.net> * update assert_block_can_be_validated comment * removes `mem` field from StateService * removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state * updates tests that use the disk to use read_service.db instead * moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service * changes `contextual_validity` to get the network from the finalized_state instead of another param * swaps out StateService with FinalizedState and NonFinalizedState in tests * adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain * removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send * makes parent_errors_map an indexmap * clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped * removes duplicate field definitions on StateService that were a result of a bad merge * update NotReadyToBeCommitted error message * Appear rustfmt * Fix doc links * Rename a function to initial_contextual_validity() * Do error tasks on Err, and success tasks on Ok * Simplify parent_error_map truncation * Rewrite best_tip() to use tip() * Rename latest_mem() to latest_non_finalized_state() ```sh fastmod latest_mem latest_non_finalized_state zebra* cargo fmt --all ``` * Simplify latest_non_finalized_state() using a new WatchReceiver API * Expand some error messages * Send the result after updating the channels, and document why * wait for chain_tip_update before cancelling download in mempool_cancel_mined * adds `sent_non_finalized_block_hashes` field to StateService * adds batched sent_hash insertions and checks sent hashes in queue_and_commit_non_finalized before adding a block to the queue * check that the `curr_buf` in SentHashes is not empty before pushing it to the `sent_bufs` * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Fix rustfmt * Check for finalized block heights using zs_contains() * adds known_utxos field to SentHashes * updates comment on SentHashes.add method * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * return early when there's a duplicate hash in QueuedBlocks.queue instead of panicking * Make finalized UTXOs near the final checkpoint available for full block verification * Replace a checkpoint height literal with the actual config * Update mainnet and testnet checkpoints - 7 October 2022 * Fix some state service init arguments * Allow more lookahead in the downloader, but less lookahead in the syncer * Add the latest config to the tests, and fix the latest config check * Increase the number of finalized blocks checked for non-finalized block UTXO spends * fix(log): reduce verbose logs for block commits (#5348) * Remove some verbose block write channel logs * Only warn about tracing endpoint if the address is actually set * Use CloneError instead of formatting a non-cloneable error Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Increase block verify timeout * Work around a known block timeout bug by using a shorter timeout Co-authored-by: teor <teor@riseup.net> Co-authored-by: Marek <mail@marek.onl> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-10-11 12:25:45 -07:00
config.consensus.clone(),
config.network.network,
);
info!("opening database, this may take a few minutes");
let (state_service, read_only_state_service, latest_chain_tip, chain_tip_change) =
change(state): Write non-finalized blocks to the state in a separate thread, to avoid network and RPC hangs (#5257) * Add a new block commit task and channels, that don't do anything yet * Add last_block_hash_sent to the state service, to avoid database accesses * Update last_block_hash_sent regardless of commit errors * Rename a field to StateService.max_queued_finalized_height * Commit finalized blocks to the state in a separate task * Check for panics in the block write task * Wait for the block commit task in tests, and check for errors * Always run a proptest that sleeps once * Add extra debugging to state shutdowns * Work around a RocksDB shutdown bug * Close the finalized block channel when we're finished with it * Only reset state queue once per error * Update some TODOs * Add a module doc comment * Drop channels and check for closed channels in the block commit task * Close state channels and tasks on drop * Remove some duplicate fields across StateService and ReadStateService * Try tweaking the shutdown steps * Update and clarify some comments * Clarify another comment * Don't try to cancel RocksDB background work on drop * Fix up some comments * Remove some duplicate code * Remove redundant workarounds for shutdown issues * Remode a redundant channel close in the block commit task * Remove a mistaken `!force` shutdown condition * Remove duplicate force-shutdown code and explain it better * Improve RPC error logging * Wait for chain tip updates in the RPC tests * Wait 2 seconds for chain tip updates before skipping them * Remove an unnecessary block_in_place() * Fix some test error messages that were changed by earlier fixes * Expand some comments, fix typos Co-authored-by: Marek <mail@marek.onl> * Actually drop children of failed blocks * Explain why we drop descendants of failed blocks * Clarify a comment * Wait for chain tip updates in a failing test on macOS * Clean duplicate finalized blocks when the non-finalized state activates * Send an error when receiving a duplicate finalized block * Update checkpoint block behaviour, document its consensus rule * Wait for chain tip changes in inbound_block_height_lookahead_limit test * Wait for the genesis block to commit in the fake peer set mempool tests * Disable unreliable mempool verification check in the send transaction test * Appease rustfmt * Use clear_finalized_block_queue() everywhere that blocks are dropped * Document how Finalized and NonFinalized clones are different * sends non-finalized blocks to the block write task * passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState * Update zebra-state/src/service/write.rs Co-authored-by: teor <teor@riseup.net> * updates comments, renames send_process_queued, other minor cleanup * update assert_block_can_be_validated comment * removes `mem` field from StateService * removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state * updates tests that use the disk to use read_service.db instead * moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service * changes `contextual_validity` to get the network from the finalized_state instead of another param * swaps out StateService with FinalizedState and NonFinalizedState in tests * adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain * removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send * makes parent_errors_map an indexmap * clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped * sends non-finalized blocks to the block write task * passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState * updates comments, renames send_process_queued, other minor cleanup * Update zebra-state/src/service/write.rs Co-authored-by: teor <teor@riseup.net> * update assert_block_can_be_validated comment * removes `mem` field from StateService * removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state * updates tests that use the disk to use read_service.db instead * moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service * changes `contextual_validity` to get the network from the finalized_state instead of another param * swaps out StateService with FinalizedState and NonFinalizedState in tests * adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain * removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send * makes parent_errors_map an indexmap * clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped * removes duplicate field definitions on StateService that were a result of a bad merge * update NotReadyToBeCommitted error message * Appear rustfmt * Fix doc links * Rename a function to initial_contextual_validity() * Do error tasks on Err, and success tasks on Ok * Simplify parent_error_map truncation * Rewrite best_tip() to use tip() * Rename latest_mem() to latest_non_finalized_state() ```sh fastmod latest_mem latest_non_finalized_state zebra* cargo fmt --all ``` * Simplify latest_non_finalized_state() using a new WatchReceiver API * Expand some error messages * Send the result after updating the channels, and document why * wait for chain_tip_update before cancelling download in mempool_cancel_mined * adds `sent_non_finalized_block_hashes` field to StateService * adds batched sent_hash insertions and checks sent hashes in queue_and_commit_non_finalized before adding a block to the queue * check that the `curr_buf` in SentHashes is not empty before pushing it to the `sent_bufs` * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Fix rustfmt * Check for finalized block heights using zs_contains() * adds known_utxos field to SentHashes * updates comment on SentHashes.add method * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * return early when there's a duplicate hash in QueuedBlocks.queue instead of panicking * Make finalized UTXOs near the final checkpoint available for full block verification * Replace a checkpoint height literal with the actual config * Update mainnet and testnet checkpoints - 7 October 2022 * Fix some state service init arguments * Allow more lookahead in the downloader, but less lookahead in the syncer * Add the latest config to the tests, and fix the latest config check * Increase the number of finalized blocks checked for non-finalized block UTXO spends * fix(log): reduce verbose logs for block commits (#5348) * Remove some verbose block write channel logs * Only warn about tracing endpoint if the address is actually set * Use CloneError instead of formatting a non-cloneable error Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Increase block verify timeout * Work around a known block timeout bug by using a shorter timeout Co-authored-by: teor <teor@riseup.net> Co-authored-by: Marek <mail@marek.onl> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-10-11 12:25:45 -07:00
zebra_state::spawn_init(
config.state.clone(),
config.network.network,
max_checkpoint_height,
config.sync.checkpoint_verify_concurrency_limit
* (VERIFICATION_PIPELINE_SCALING_MULTIPLIER + 1),
)
.await?;
let state = ServiceBuilder::new()
.buffer(Self::state_buffer_bound())
.service(state_service);
info!("initializing network");
// The service that our node uses to respond to requests by peers. The
// load_shed middleware ensures that we reduce the size of the peer set
// in response to excess load.
//
// # Security
//
// This layer stack is security-sensitive, modifying it can cause hangs,
// or enable denial of service attacks.
//
// See `zebra_network::Connection::drive_peer_request()` for details.
let (setup_tx, setup_rx) = oneshot::channel();
let inbound = ServiceBuilder::new()
.load_shed()
.buffer(inbound::downloads::MAX_INBOUND_CONCURRENCY)
.timeout(MAX_INBOUND_RESPONSE_TIME)
.service(Inbound::new(
config.sync.full_verify_concurrency_limit,
setup_rx,
));
let (peer_set, address_book) = zebra_network::init(
config.network.clone(),
inbound,
latest_chain_tip.clone(),
user_agent(),
)
.await;
info!("initializing verifiers");
let (block_verifier_router, tx_verifier, consensus_task_handles, max_checkpoint_height) =
zebra_consensus::router::init(
config.consensus.clone(),
config.network.network,
state.clone(),
)
.await;
info!("initializing syncer");
let (syncer, sync_status) = ChainSync::new(
&config,
max_checkpoint_height,
peer_set.clone(),
block_verifier_router.clone(),
state.clone(),
latest_chain_tip.clone(),
);
info!("initializing mempool");
let (mempool, mempool_transaction_receiver) = Mempool::new(
&config.mempool,
peer_set.clone(),
state.clone(),
tx_verifier,
sync_status.clone(),
latest_chain_tip.clone(),
chain_tip_change.clone(),
);
let mempool = BoxService::new(mempool);
let mempool = ServiceBuilder::new()
.buffer(mempool::downloads::MAX_INBOUND_CONCURRENCY)
.service(mempool);
info!("fully initializing inbound peer request handler");
// Fully start the inbound service as soon as possible
let setup_data = InboundSetupData {
address_book: address_book.clone(),
block_download_peer_set: peer_set.clone(),
block_verifier: block_verifier_router.clone(),
mempool: mempool.clone(),
state: state.clone(),
latest_chain_tip: latest_chain_tip.clone(),
};
setup_tx
.send(setup_data)
.map_err(|_| eyre!("could not send setup data to inbound service"))?;
// And give it time to clear its queue
tokio::task::yield_now().await;
#[cfg(not(feature = "getblocktemplate-rpcs"))]
if config.mining != zebra_rpc::config::mining::Config::default() {
warn!(
"Unused mining section in config,\
compile with 'getblocktemplate-rpcs' feature to use mining RPCs"
);
}
// Launch RPC server
info!("spawning RPC server");
let (rpc_task_handle, rpc_tx_queue_task_handle, rpc_server) = RpcServer::spawn(
fix(zebrad): accept default subcommand arguments and print consistent usage information for top-level 'help' subcommand (#6801) * updates Cargo.toml * Migrate to abscissa 0.7.0 * Avoid panic from calling color_eyre::install twice * Uses 'start' as the default subcommand * updates default cmd logic * Fixes minor cli issues * removes outdated check in acceptance test * Adds a test for process_cli_args, fixes version_args test. Adds -V to process_cli_args match case * Revert "fix(clippy): Silence future-incompat warnings until we upgrade Abscissa (#6024)" This reverts commit dd90f79b4824af6b1ae22ee69abdf0f605da84cd. * Drops the worker guard to flush logs when zebra shuts down * Adds cargo feature to clap * restores process_cli_args * updates deny.toml * Updates EntryPoint help template * Updates subcommand help msgs * removes trailing whitespace, capitalizes sentences * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * revert parts of revert "Revert fix(clippy): Silence future-incompat warnings until we upgrade Abscissa" * Applies suggestions from code review * Moves EntryPoint to its own module * fixes version_args test * Updates changelog * Prunes redundant test cases * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Revert "Prunes redundant test cases" This reverts commit 3f7397918489144805c17d0594775aa699e87b9d. * Update zebrad/src/commands/entry_point.rs Co-authored-by: teor <teor@riseup.net> * Add missing import * Updates `process_cli_args` to return a result --------- Co-authored-by: teor <teor@riseup.net>
2023-06-06 23:03:42 -07:00
config.rpc.clone(),
config.mining.clone(),
build_version(),
user_agent(),
feature(rpc): implement getblock api call (#3707) * feature(rpc): start adding a `getblock` method * fix(rpc): replace oneshot * fix(rpc): replace a panic with error * fix(rpc): fix test * feature(rpc): add hex to response * refactor(rpc): use generic instead of alias * docs(rpc): improve docs for getblock method * test(rpc): add a test for getblock method * deps(rpc): remove non needed tower features Co-authored-by: teor <teor@riseup.net> * docs(rpc): add a note to getblock doc * refactor(rpc): replace alias * fix(rpc): use `zcash_serialize_to_vec()` instead of logging format * tests(rpc): add network argument to `populated_state()` * refactor(rpc): use an error for state service readiness * fix(rpc): add parameter * fix(rpc): clippy * nit(rpc): remove new line from imports * fix(rpc): remove commented code * fix(rpc): simplify error Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Use a `SerializedBlock` type to help serializing blocks (#3725) * Create a `SerializedBlock` helper type Create a type that can be used as a byte slice, but is guaranteed to represent a valid block. * Use `into_iter` instead of `iter` There's no need to borrow the elements, they can be moved out directly. This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`, so a `SerializedBlock` can't be built directly from an `&Arc<Block>`. * Use `SerializedBlock` in `GetBlock` Make the type stricter to avoid storing possibly invalid values. The bytes are still serialized as a hexadecimal string, through the usage of `hex`. The `serde::Deserialize` can't be derived because `hex` requires the type to also implement `FromHex`. * feature(rpc): add suggestions from code review Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * tests(rpc): make sure mempool has no requests in get_block test * fix(rpc): change height argument type in getblock method * fix(rpc): rustfmt * fix(rpc): replace panic * fix(rpc): change getblock response * fix(rpc): fix lightwalletd test * tests(rpc): add a getblock error test * fix(rpc): try another regex Co-authored-by: teor <teor@riseup.net> Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2022-03-09 17:12:41 -08:00
mempool.clone(),
feat(mine): Add an internal Zcash miner to Zebra (#8136) * Patch equihash to use the solver branch * Add an internal-miner feature and set up its dependencies * Remove 'Experimental' from mining RPC docs * Fix a nightly clippy::question_mark lint * Move a byte array utility function to zebra-chain * fixup! Add an internal-miner feature and set up its dependencies * Add an equihash::Solution::solve() method with difficulty checks * Check solution is valid before returning it * Add a TODO to check for peers before mining * Move config validation into GetBlockTemplateRpcImpl::new() * fixup! fixup! Add an internal-miner feature and set up its dependencies * Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls * Start adding an internal miner component * Add the miner task to the start command * Add basic miner code * Split out a method to mine one block * Spawn to a blocking thread * Wait until a valid template is available * Handle shutdown * Run mining on low priority threads * Ignore some invalid solutions * Use a difference nonce for each solver thread * Update TODOs * Change the patch into a renamed dependency to simplify crate releases * Clean up instrumentation and TODOs * Make RPC instances cloneable and clean up generics * Make LongPollId Copy so it's easier to use * Add API to restart mining if there's a new block template * Actually restart mining if there's a new block template * Tidy instrumentation * fixup! Move config validation into GetBlockTemplateRpcImpl::new() * fixup! Make RPC instances cloneable and clean up generics * Run the template generator and one miner concurrently * Reduce logging * Fix a bug in getblocktemplate RPC tip change detection * Work around some watch channel change bugs * Rate-limit template changes in the receiver * Run one mining solver per available core * Use updated C code with double-free protection * Update to the latest solver branch * Return and submit all valid solutions * Document what INPUT_LENGTH means * Fix watch channel change detection * Don't return early when a mining task fails * Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness * Make existing parallelism docs and configs consistent * Add a mining parallelism config * Use the minimum of the configured or available threads for mining * Ignore optional feature fields in tests * Downgrade some frequent logs to debug * Document new zebrad features and tasks * Describe the internal-miner feature in the CHANGELOG * Update dependency to de-duplicate equihash solutions * Use futures::StreamExt instead of TryStreamExt * Fix a panic message typo
2024-01-11 06:41:01 -08:00
read_only_state_service.clone(),
block_verifier_router.clone(),
sync_status.clone(),
feat(mine): Add an internal Zcash miner to Zebra (#8136) * Patch equihash to use the solver branch * Add an internal-miner feature and set up its dependencies * Remove 'Experimental' from mining RPC docs * Fix a nightly clippy::question_mark lint * Move a byte array utility function to zebra-chain * fixup! Add an internal-miner feature and set up its dependencies * Add an equihash::Solution::solve() method with difficulty checks * Check solution is valid before returning it * Add a TODO to check for peers before mining * Move config validation into GetBlockTemplateRpcImpl::new() * fixup! fixup! Add an internal-miner feature and set up its dependencies * Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls * Start adding an internal miner component * Add the miner task to the start command * Add basic miner code * Split out a method to mine one block * Spawn to a blocking thread * Wait until a valid template is available * Handle shutdown * Run mining on low priority threads * Ignore some invalid solutions * Use a difference nonce for each solver thread * Update TODOs * Change the patch into a renamed dependency to simplify crate releases * Clean up instrumentation and TODOs * Make RPC instances cloneable and clean up generics * Make LongPollId Copy so it's easier to use * Add API to restart mining if there's a new block template * Actually restart mining if there's a new block template * Tidy instrumentation * fixup! Move config validation into GetBlockTemplateRpcImpl::new() * fixup! Make RPC instances cloneable and clean up generics * Run the template generator and one miner concurrently * Reduce logging * Fix a bug in getblocktemplate RPC tip change detection * Work around some watch channel change bugs * Rate-limit template changes in the receiver * Run one mining solver per available core * Use updated C code with double-free protection * Update to the latest solver branch * Return and submit all valid solutions * Document what INPUT_LENGTH means * Fix watch channel change detection * Don't return early when a mining task fails * Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness * Make existing parallelism docs and configs consistent * Add a mining parallelism config * Use the minimum of the configured or available threads for mining * Ignore optional feature fields in tests * Downgrade some frequent logs to debug * Document new zebrad features and tasks * Describe the internal-miner feature in the CHANGELOG * Update dependency to de-duplicate equihash solutions * Use futures::StreamExt instead of TryStreamExt * Fix a panic message typo
2024-01-11 06:41:01 -08:00
address_book.clone(),
latest_chain_tip.clone(),
config.network.network,
feature(rpc): implement getblock api call (#3707) * feature(rpc): start adding a `getblock` method * fix(rpc): replace oneshot * fix(rpc): replace a panic with error * fix(rpc): fix test * feature(rpc): add hex to response * refactor(rpc): use generic instead of alias * docs(rpc): improve docs for getblock method * test(rpc): add a test for getblock method * deps(rpc): remove non needed tower features Co-authored-by: teor <teor@riseup.net> * docs(rpc): add a note to getblock doc * refactor(rpc): replace alias * fix(rpc): use `zcash_serialize_to_vec()` instead of logging format * tests(rpc): add network argument to `populated_state()` * refactor(rpc): use an error for state service readiness * fix(rpc): add parameter * fix(rpc): clippy * nit(rpc): remove new line from imports * fix(rpc): remove commented code * fix(rpc): simplify error Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * Use a `SerializedBlock` type to help serializing blocks (#3725) * Create a `SerializedBlock` helper type Create a type that can be used as a byte slice, but is guaranteed to represent a valid block. * Use `into_iter` instead of `iter` There's no need to borrow the elements, they can be moved out directly. This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`, so a `SerializedBlock` can't be built directly from an `&Arc<Block>`. * Use `SerializedBlock` in `GetBlock` Make the type stricter to avoid storing possibly invalid values. The bytes are still serialized as a hexadecimal string, through the usage of `hex`. The `serde::Deserialize` can't be derived because `hex` requires the type to also implement `FromHex`. * feature(rpc): add suggestions from code review Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com> * tests(rpc): make sure mempool has no requests in get_block test * fix(rpc): change height argument type in getblock method * fix(rpc): rustfmt * fix(rpc): replace panic * fix(rpc): change getblock response * fix(rpc): fix lightwalletd test * tests(rpc): add a getblock error test * fix(rpc): try another regex Co-authored-by: teor <teor@riseup.net> Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2022-03-09 17:12:41 -08:00
);
// Start concurrent tasks which don't add load to other tasks
info!("spawning block gossip task");
let block_gossip_task_handle = tokio::spawn(
sync::gossip_best_tip_block_hashes(
sync_status.clone(),
chain_tip_change.clone(),
peer_set.clone(),
)
.in_current_span(),
);
info!("spawning mempool queue checker task");
Refactor `SentTransactionHash` to be a stricter type (#3706) * Stub `sendrawtransaction` RPC method Register the RPC method, and stub an implementation that currently just panics. The method has a single `String` parameter with the hexadecimal string of the raw transaction's bytes and returns a `SentTransactionHash` wrapper type that's just a hexadecimal `String` of the sent transaction's hash. * Add mempool service instance to `RpcImpl` Use a type parameter to represent the mempool service using the interface defined by `zebra-node-services`. * Update test vector to use a mock mempool service Update the test to be compatible with the changes to `RpcImpl`. The mock mempool service is expected to not be used during the test. * Use a `tower::Buffer` for the mempool service Make it simpler to send requests to the service in a concurrent manner. * Return a `Future` from `send_raw_transaction` Make the call asynchronous. * Implement `sendrawtransaction` RPC Deserialize the transaction and send it to be queued for verification and subsequent inclusion in the mempool. * Test if mempool receives sent raw transaction Use a mock service as the mempool service and check that it receives a sent raw transaction. * Test using non-hexadecimal string parameter The method should return an error. * Test with bytes that fail deserialization Check that the method returns an invalid parameters error if the input can't be deserialized as a `Transaction`. * Test if mempool errors are forwarded to caller Mempool service errors should be sent back to the remote caller as server errors. * Test transactions rejected by the mempool service Transactions that are rejected by the mempool service should result in a server error being sent to the caller. * Improve error message Add the word "structurally" to make it clear that the issue is in the transaction's deserialization. Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com> * Add note regarding missing `allowhighfees` param. The parameter isn't supported yet because `lightwalletd` doesn't use it. * Update the documentation to be consistent Follow the convention adopted by the `get_info` RPC method. * Implement `ToHex` and `FromHex` for `Hash` Make it easier to generate hexadecimal strings from `transaction::Hash` instances. * Use `ToHex` in `Debug` and `Display` Reduce repeated code. * Refactor to add `bytes_in_display_order` method Use it to remove repeated code and improve clarity a bit. * Use `hex::serialize` to serialize transaction hash Make the type stricter in its contents, while still serializing the transaction has as a hexadecimal string. * Simplify serialization attribute Deserialization should also use `hex::deserialize`, so using the shorter attribute makes things easier to read and more future proof. * Update zebra-chain/src/transaction/hash.rs * Remove unnecessary lifetime The anonymous lifetime is automatically inferred by the compiler. Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
2022-03-08 01:14:21 -08:00
let mempool_queue_checker_task_handle = mempool::QueueChecker::spawn(mempool.clone());
info!("spawning mempool transaction gossip task");
let tx_gossip_task_handle = tokio::spawn(
mempool::gossip_mempool_transaction_id(mempool_transaction_receiver, peer_set.clone())
.in_current_span(),
);
info!("spawning delete old databases task");
let mut old_databases_task_handle = zebra_state::check_and_delete_old_state_databases(
&config.state,
config.network.network,
);
info!("spawning progress logging task");
let progress_task_handle = tokio::spawn(
show_block_chain_progress(
config.network.network,
latest_chain_tip.clone(),
sync_status.clone(),
)
.in_current_span(),
);
// Spawn never ending end of support task.
info!("spawning end of support checking task");
let end_of_support_task_handle = tokio::spawn(
feat(mine): Add an internal Zcash miner to Zebra (#8136) * Patch equihash to use the solver branch * Add an internal-miner feature and set up its dependencies * Remove 'Experimental' from mining RPC docs * Fix a nightly clippy::question_mark lint * Move a byte array utility function to zebra-chain * fixup! Add an internal-miner feature and set up its dependencies * Add an equihash::Solution::solve() method with difficulty checks * Check solution is valid before returning it * Add a TODO to check for peers before mining * Move config validation into GetBlockTemplateRpcImpl::new() * fixup! fixup! Add an internal-miner feature and set up its dependencies * Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls * Start adding an internal miner component * Add the miner task to the start command * Add basic miner code * Split out a method to mine one block * Spawn to a blocking thread * Wait until a valid template is available * Handle shutdown * Run mining on low priority threads * Ignore some invalid solutions * Use a difference nonce for each solver thread * Update TODOs * Change the patch into a renamed dependency to simplify crate releases * Clean up instrumentation and TODOs * Make RPC instances cloneable and clean up generics * Make LongPollId Copy so it's easier to use * Add API to restart mining if there's a new block template * Actually restart mining if there's a new block template * Tidy instrumentation * fixup! Move config validation into GetBlockTemplateRpcImpl::new() * fixup! Make RPC instances cloneable and clean up generics * Run the template generator and one miner concurrently * Reduce logging * Fix a bug in getblocktemplate RPC tip change detection * Work around some watch channel change bugs * Rate-limit template changes in the receiver * Run one mining solver per available core * Use updated C code with double-free protection * Update to the latest solver branch * Return and submit all valid solutions * Document what INPUT_LENGTH means * Fix watch channel change detection * Don't return early when a mining task fails * Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness * Make existing parallelism docs and configs consistent * Add a mining parallelism config * Use the minimum of the configured or available threads for mining * Ignore optional feature fields in tests * Downgrade some frequent logs to debug * Document new zebrad features and tasks * Describe the internal-miner feature in the CHANGELOG * Update dependency to de-duplicate equihash solutions * Use futures::StreamExt instead of TryStreamExt * Fix a panic message typo
2024-01-11 06:41:01 -08:00
sync::end_of_support::start(config.network.network, latest_chain_tip.clone())
.in_current_span(),
);
// Give the inbound service more time to clear its queue,
// then start concurrent tasks that can add load to the inbound service
// (by opening more peer connections, so those peers send us requests)
tokio::task::yield_now().await;
// The crawler only activates immediately in tests that use mempool debug mode
info!("spawning mempool crawler task");
let mempool_crawler_task_handle = mempool::Crawler::spawn(
&config.mempool,
peer_set,
mempool.clone(),
feat(mine): Add an internal Zcash miner to Zebra (#8136) * Patch equihash to use the solver branch * Add an internal-miner feature and set up its dependencies * Remove 'Experimental' from mining RPC docs * Fix a nightly clippy::question_mark lint * Move a byte array utility function to zebra-chain * fixup! Add an internal-miner feature and set up its dependencies * Add an equihash::Solution::solve() method with difficulty checks * Check solution is valid before returning it * Add a TODO to check for peers before mining * Move config validation into GetBlockTemplateRpcImpl::new() * fixup! fixup! Add an internal-miner feature and set up its dependencies * Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls * Start adding an internal miner component * Add the miner task to the start command * Add basic miner code * Split out a method to mine one block * Spawn to a blocking thread * Wait until a valid template is available * Handle shutdown * Run mining on low priority threads * Ignore some invalid solutions * Use a difference nonce for each solver thread * Update TODOs * Change the patch into a renamed dependency to simplify crate releases * Clean up instrumentation and TODOs * Make RPC instances cloneable and clean up generics * Make LongPollId Copy so it's easier to use * Add API to restart mining if there's a new block template * Actually restart mining if there's a new block template * Tidy instrumentation * fixup! Move config validation into GetBlockTemplateRpcImpl::new() * fixup! Make RPC instances cloneable and clean up generics * Run the template generator and one miner concurrently * Reduce logging * Fix a bug in getblocktemplate RPC tip change detection * Work around some watch channel change bugs * Rate-limit template changes in the receiver * Run one mining solver per available core * Use updated C code with double-free protection * Update to the latest solver branch * Return and submit all valid solutions * Document what INPUT_LENGTH means * Fix watch channel change detection * Don't return early when a mining task fails * Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness * Make existing parallelism docs and configs consistent * Add a mining parallelism config * Use the minimum of the configured or available threads for mining * Ignore optional feature fields in tests * Downgrade some frequent logs to debug * Document new zebrad features and tasks * Describe the internal-miner feature in the CHANGELOG * Update dependency to de-duplicate equihash solutions * Use futures::StreamExt instead of TryStreamExt * Fix a panic message typo
2024-01-11 06:41:01 -08:00
sync_status.clone(),
chain_tip_change.clone(),
);
info!("spawning syncer task");
let syncer_task_handle = tokio::spawn(syncer.sync().in_current_span());
#[cfg(feature = "shielded-scan")]
// Spawn never ending scan task only if we have keys to scan for.
let scan_task_handle = if !config.shielded_scan.sapling_keys_to_scan.is_empty() {
// TODO: log the number of keys and update the scan_task_starts() test
info!("spawning shielded scanner with configured viewing keys");
zebra_scan::spawn_init(
&config.shielded_scan,
config.network.network,
state,
chain_tip_change,
)
} else {
tokio::spawn(std::future::pending().in_current_span())
};
#[cfg(not(feature = "shielded-scan"))]
// Spawn a dummy scan task which doesn't do anything and never finishes.
let scan_task_handle: tokio::task::JoinHandle<Result<(), Report>> =
tokio::spawn(std::future::pending().in_current_span());
feat(mine): Add an internal Zcash miner to Zebra (#8136) * Patch equihash to use the solver branch * Add an internal-miner feature and set up its dependencies * Remove 'Experimental' from mining RPC docs * Fix a nightly clippy::question_mark lint * Move a byte array utility function to zebra-chain * fixup! Add an internal-miner feature and set up its dependencies * Add an equihash::Solution::solve() method with difficulty checks * Check solution is valid before returning it * Add a TODO to check for peers before mining * Move config validation into GetBlockTemplateRpcImpl::new() * fixup! fixup! Add an internal-miner feature and set up its dependencies * Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls * Start adding an internal miner component * Add the miner task to the start command * Add basic miner code * Split out a method to mine one block * Spawn to a blocking thread * Wait until a valid template is available * Handle shutdown * Run mining on low priority threads * Ignore some invalid solutions * Use a difference nonce for each solver thread * Update TODOs * Change the patch into a renamed dependency to simplify crate releases * Clean up instrumentation and TODOs * Make RPC instances cloneable and clean up generics * Make LongPollId Copy so it's easier to use * Add API to restart mining if there's a new block template * Actually restart mining if there's a new block template * Tidy instrumentation * fixup! Move config validation into GetBlockTemplateRpcImpl::new() * fixup! Make RPC instances cloneable and clean up generics * Run the template generator and one miner concurrently * Reduce logging * Fix a bug in getblocktemplate RPC tip change detection * Work around some watch channel change bugs * Rate-limit template changes in the receiver * Run one mining solver per available core * Use updated C code with double-free protection * Update to the latest solver branch * Return and submit all valid solutions * Document what INPUT_LENGTH means * Fix watch channel change detection * Don't return early when a mining task fails * Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness * Make existing parallelism docs and configs consistent * Add a mining parallelism config * Use the minimum of the configured or available threads for mining * Ignore optional feature fields in tests * Downgrade some frequent logs to debug * Document new zebrad features and tasks * Describe the internal-miner feature in the CHANGELOG * Update dependency to de-duplicate equihash solutions * Use futures::StreamExt instead of TryStreamExt * Fix a panic message typo
2024-01-11 06:41:01 -08:00
// And finally, spawn the internal Zcash miner, if it is enabled.
//
// TODO: add a config to enable the miner rather than a feature.
#[cfg(feature = "internal-miner")]
let miner_task_handle = if config.mining.is_internal_miner_enabled() {
info!("spawning Zcash miner");
let rpc = zebra_rpc::methods::get_block_template_rpcs::GetBlockTemplateRpcImpl::new(
config.network.network,
config.mining.clone(),
mempool,
read_only_state_service,
latest_chain_tip,
block_verifier_router,
sync_status,
address_book,
);
crate::components::miner::spawn_init(&config.mining, rpc)
} else {
tokio::spawn(std::future::pending().in_current_span())
};
#[cfg(not(feature = "internal-miner"))]
// Spawn a dummy miner task which doesn't do anything and never finishes.
let miner_task_handle: tokio::task::JoinHandle<Result<(), Report>> =
tokio::spawn(std::future::pending().in_current_span());
info!("spawned initial Zebra tasks");
// TODO: put tasks into an ongoing FuturesUnordered and a startup FuturesUnordered?
// ongoing tasks
pin!(rpc_task_handle);
pin!(rpc_tx_queue_task_handle);
pin!(syncer_task_handle);
pin!(block_gossip_task_handle);
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
pin!(mempool_crawler_task_handle);
pin!(mempool_queue_checker_task_handle);
pin!(tx_gossip_task_handle);
pin!(progress_task_handle);
pin!(end_of_support_task_handle);
pin!(scan_task_handle);
feat(mine): Add an internal Zcash miner to Zebra (#8136) * Patch equihash to use the solver branch * Add an internal-miner feature and set up its dependencies * Remove 'Experimental' from mining RPC docs * Fix a nightly clippy::question_mark lint * Move a byte array utility function to zebra-chain * fixup! Add an internal-miner feature and set up its dependencies * Add an equihash::Solution::solve() method with difficulty checks * Check solution is valid before returning it * Add a TODO to check for peers before mining * Move config validation into GetBlockTemplateRpcImpl::new() * fixup! fixup! Add an internal-miner feature and set up its dependencies * Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls * Start adding an internal miner component * Add the miner task to the start command * Add basic miner code * Split out a method to mine one block * Spawn to a blocking thread * Wait until a valid template is available * Handle shutdown * Run mining on low priority threads * Ignore some invalid solutions * Use a difference nonce for each solver thread * Update TODOs * Change the patch into a renamed dependency to simplify crate releases * Clean up instrumentation and TODOs * Make RPC instances cloneable and clean up generics * Make LongPollId Copy so it's easier to use * Add API to restart mining if there's a new block template * Actually restart mining if there's a new block template * Tidy instrumentation * fixup! Move config validation into GetBlockTemplateRpcImpl::new() * fixup! Make RPC instances cloneable and clean up generics * Run the template generator and one miner concurrently * Reduce logging * Fix a bug in getblocktemplate RPC tip change detection * Work around some watch channel change bugs * Rate-limit template changes in the receiver * Run one mining solver per available core * Use updated C code with double-free protection * Update to the latest solver branch * Return and submit all valid solutions * Document what INPUT_LENGTH means * Fix watch channel change detection * Don't return early when a mining task fails * Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness * Make existing parallelism docs and configs consistent * Add a mining parallelism config * Use the minimum of the configured or available threads for mining * Ignore optional feature fields in tests * Downgrade some frequent logs to debug * Document new zebrad features and tasks * Describe the internal-miner feature in the CHANGELOG * Update dependency to de-duplicate equihash solutions * Use futures::StreamExt instead of TryStreamExt * Fix a panic message typo
2024-01-11 06:41:01 -08:00
pin!(miner_task_handle);
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
// startup tasks
let BackgroundTaskHandles {
mut state_checkpoint_verify_handle,
} = consensus_task_handles;
let state_checkpoint_verify_handle_fused = (&mut state_checkpoint_verify_handle).fuse();
pin!(state_checkpoint_verify_handle_fused);
let old_databases_task_handle_fused = (&mut old_databases_task_handle).fuse();
pin!(old_databases_task_handle_fused);
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
// Wait for tasks to finish
let exit_status = loop {
let mut exit_when_task_finishes = true;
let result = select! {
rpc_result = &mut rpc_task_handle => {
rpc_result
.expect("unexpected panic in the rpc task");
info!("rpc task exited");
Ok(())
}
rpc_tx_queue_result = &mut rpc_tx_queue_task_handle => {
rpc_tx_queue_result
.expect("unexpected panic in the rpc transaction queue task");
info!("rpc transaction queue task exited");
Ok(())
}
sync_result = &mut syncer_task_handle => sync_result
.expect("unexpected panic in the syncer task")
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
.map(|_| info!("syncer task exited")),
block_gossip_result = &mut block_gossip_task_handle => block_gossip_result
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
.expect("unexpected panic in the chain tip block gossip task")
.map(|_| info!("chain tip block gossip task exited"))
.map_err(|e| eyre!(e)),
mempool_crawl_result = &mut mempool_crawler_task_handle => mempool_crawl_result
.expect("unexpected panic in the mempool crawler")
.map(|_| info!("mempool crawler task exited"))
.map_err(|e| eyre!(e)),
mempool_queue_result = &mut mempool_queue_checker_task_handle => mempool_queue_result
.expect("unexpected panic in the mempool queue checker")
.map(|_| info!("mempool queue checker task exited"))
.map_err(|e| eyre!(e)),
tx_gossip_result = &mut tx_gossip_task_handle => tx_gossip_result
.expect("unexpected panic in the transaction gossip task")
.map(|_| info!("transaction gossip task exited"))
.map_err(|e| eyre!(e)),
feat(ui): Add a terminal-based progress bar to Zebra (#6235) * Implement Display and to_string() for NetworkUpgrade * Add a progress-bar feature to zebrad * Add the progress bar writer to the tracing component * Add a block progress bar transmitter * Correctly shut down the progress bar, and shut it down on an interrupt * Make it clearer that the progress task never exits * Add a config for writing logs to a file * Add a progress-bar feature to zebra-network * Add a progress bar for the address book size * Add progress bars for never attempted and failed peers * Add an optional limit and label to connection counters * Add open connection progress bars * Improve CheckpointList API and CheckpointVerifier debugging * Add checkpoint index and checkpoint queue progress bars * Security: Limit the number of non-finalized chains tracked by Zebra * Make some NonFinalizedState methods available with proptest-impl * Add a non-finalized chain count progress bar * Track the last fork height for newly forked chains * Add a should_count_metrics to Chain * Add a display method for PartialCumulativeWork * Add a progress bar for each chain fork * Add a NonFinalizedState::disable_metrics() method and switch to using it * Move metrics out of Chain because we can't update Arc<Chain> * Fix: consistently use best chain order when searching chains * Track Chain progress bars in NonFinalizedState * Display work as bits, not a multiple of the target difficulty * Handle negative fork lengths by reporting "No fork" * Correctly disable unused fork bars * clippy: rewrite using `match _.cmp(_) { ... }` * Initial mempool progress bar implementation * Update Cargo.lock * Add the actual transaction size as a description to the cost bar * Only show mempool progress bars after first activation * Add queued and rejected mempool progress bars * Clarify cost note is actual size * Add tracing.log_file config and progress-bar feature to zebrad docs * Derive Clone for Chain * Upgrade to howudoin 0.1.2 and remove some bug workarounds * Directly call the debug formatter to Display a Network Co-authored-by: Arya <aryasolhi@gmail.com> * Rename the address count metric to num_addresses Co-authored-by: Arya <aryasolhi@gmail.com> * Simplify reverse checkpoint lookup Co-authored-by: Arya <aryasolhi@gmail.com> * Simplify progress bar shutdown code Co-authored-by: Arya <aryasolhi@gmail.com> * Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE * Document that the progress task runs forever * Fix progress log formatting * If progress-bar is on, log to a file by default * Create missing directories for log files * Add file security docs for running Zebra with elevated permissions * Document automatic log file, spell progress-bar correctly --------- Co-authored-by: Arya <aryasolhi@gmail.com>
2023-04-13 01:42:17 -07:00
// The progress task runs forever, unless it panics.
// So we don't need to provide an exit status for it.
progress_result = &mut progress_task_handle => {
feat(ui): Add a terminal-based progress bar to Zebra (#6235) * Implement Display and to_string() for NetworkUpgrade * Add a progress-bar feature to zebrad * Add the progress bar writer to the tracing component * Add a block progress bar transmitter * Correctly shut down the progress bar, and shut it down on an interrupt * Make it clearer that the progress task never exits * Add a config for writing logs to a file * Add a progress-bar feature to zebra-network * Add a progress bar for the address book size * Add progress bars for never attempted and failed peers * Add an optional limit and label to connection counters * Add open connection progress bars * Improve CheckpointList API and CheckpointVerifier debugging * Add checkpoint index and checkpoint queue progress bars * Security: Limit the number of non-finalized chains tracked by Zebra * Make some NonFinalizedState methods available with proptest-impl * Add a non-finalized chain count progress bar * Track the last fork height for newly forked chains * Add a should_count_metrics to Chain * Add a display method for PartialCumulativeWork * Add a progress bar for each chain fork * Add a NonFinalizedState::disable_metrics() method and switch to using it * Move metrics out of Chain because we can't update Arc<Chain> * Fix: consistently use best chain order when searching chains * Track Chain progress bars in NonFinalizedState * Display work as bits, not a multiple of the target difficulty * Handle negative fork lengths by reporting "No fork" * Correctly disable unused fork bars * clippy: rewrite using `match _.cmp(_) { ... }` * Initial mempool progress bar implementation * Update Cargo.lock * Add the actual transaction size as a description to the cost bar * Only show mempool progress bars after first activation * Add queued and rejected mempool progress bars * Clarify cost note is actual size * Add tracing.log_file config and progress-bar feature to zebrad docs * Derive Clone for Chain * Upgrade to howudoin 0.1.2 and remove some bug workarounds * Directly call the debug formatter to Display a Network Co-authored-by: Arya <aryasolhi@gmail.com> * Rename the address count metric to num_addresses Co-authored-by: Arya <aryasolhi@gmail.com> * Simplify reverse checkpoint lookup Co-authored-by: Arya <aryasolhi@gmail.com> * Simplify progress bar shutdown code Co-authored-by: Arya <aryasolhi@gmail.com> * Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE * Document that the progress task runs forever * Fix progress log formatting * If progress-bar is on, log to a file by default * Create missing directories for log files * Add file security docs for running Zebra with elevated permissions * Document automatic log file, spell progress-bar correctly --------- Co-authored-by: Arya <aryasolhi@gmail.com>
2023-04-13 01:42:17 -07:00
info!("chain progress task exited");
progress_result
.expect("unexpected panic in the chain progress task");
}
end_of_support_result = &mut end_of_support_task_handle => end_of_support_result
.expect("unexpected panic in the end of support task")
.map(|_| info!("end of support task exited")),
// We also expect the state checkpoint verify task to finish.
state_checkpoint_verify_result = &mut state_checkpoint_verify_handle_fused => {
state_checkpoint_verify_result
.unwrap_or_else(|_| panic!(
"unexpected panic checking previous state followed the best chain"));
exit_when_task_finishes = false;
Ok(())
}
// And the old databases task should finish while Zebra is running.
old_databases_result = &mut old_databases_task_handle_fused => {
old_databases_result
.unwrap_or_else(|_| panic!(
"unexpected panic deleting old database directories"));
exit_when_task_finishes = false;
Ok(())
}
scan_result = &mut scan_task_handle => scan_result
.expect("unexpected panic in the scan task")
.map(|_| info!("scan task exited")),
feat(mine): Add an internal Zcash miner to Zebra (#8136) * Patch equihash to use the solver branch * Add an internal-miner feature and set up its dependencies * Remove 'Experimental' from mining RPC docs * Fix a nightly clippy::question_mark lint * Move a byte array utility function to zebra-chain * fixup! Add an internal-miner feature and set up its dependencies * Add an equihash::Solution::solve() method with difficulty checks * Check solution is valid before returning it * Add a TODO to check for peers before mining * Move config validation into GetBlockTemplateRpcImpl::new() * fixup! fixup! Add an internal-miner feature and set up its dependencies * Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls * Start adding an internal miner component * Add the miner task to the start command * Add basic miner code * Split out a method to mine one block * Spawn to a blocking thread * Wait until a valid template is available * Handle shutdown * Run mining on low priority threads * Ignore some invalid solutions * Use a difference nonce for each solver thread * Update TODOs * Change the patch into a renamed dependency to simplify crate releases * Clean up instrumentation and TODOs * Make RPC instances cloneable and clean up generics * Make LongPollId Copy so it's easier to use * Add API to restart mining if there's a new block template * Actually restart mining if there's a new block template * Tidy instrumentation * fixup! Move config validation into GetBlockTemplateRpcImpl::new() * fixup! Make RPC instances cloneable and clean up generics * Run the template generator and one miner concurrently * Reduce logging * Fix a bug in getblocktemplate RPC tip change detection * Work around some watch channel change bugs * Rate-limit template changes in the receiver * Run one mining solver per available core * Use updated C code with double-free protection * Update to the latest solver branch * Return and submit all valid solutions * Document what INPUT_LENGTH means * Fix watch channel change detection * Don't return early when a mining task fails * Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness * Make existing parallelism docs and configs consistent * Add a mining parallelism config * Use the minimum of the configured or available threads for mining * Ignore optional feature fields in tests * Downgrade some frequent logs to debug * Document new zebrad features and tasks * Describe the internal-miner feature in the CHANGELOG * Update dependency to de-duplicate equihash solutions * Use futures::StreamExt instead of TryStreamExt * Fix a panic message typo
2024-01-11 06:41:01 -08:00
miner_result = &mut miner_task_handle => miner_result
.expect("unexpected panic in the miner task")
.map(|_| info!("miner task exited")),
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
};
// Stop Zebra if a task finished and returned an error,
// or if an ongoing task exited.
if let Err(err) = result {
break Err(err);
}
if exit_when_task_finishes {
break Ok(());
}
};
info!("exiting Zebra because an ongoing task exited: asking other tasks to stop");
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
// ongoing tasks
rpc_task_handle.abort();
rpc_tx_queue_task_handle.abort();
syncer_task_handle.abort();
block_gossip_task_handle.abort();
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
mempool_crawler_task_handle.abort();
mempool_queue_checker_task_handle.abort();
tx_gossip_task_handle.abort();
progress_task_handle.abort();
end_of_support_task_handle.abort();
scan_task_handle.abort();
feat(mine): Add an internal Zcash miner to Zebra (#8136) * Patch equihash to use the solver branch * Add an internal-miner feature and set up its dependencies * Remove 'Experimental' from mining RPC docs * Fix a nightly clippy::question_mark lint * Move a byte array utility function to zebra-chain * fixup! Add an internal-miner feature and set up its dependencies * Add an equihash::Solution::solve() method with difficulty checks * Check solution is valid before returning it * Add a TODO to check for peers before mining * Move config validation into GetBlockTemplateRpcImpl::new() * fixup! fixup! Add an internal-miner feature and set up its dependencies * Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls * Start adding an internal miner component * Add the miner task to the start command * Add basic miner code * Split out a method to mine one block * Spawn to a blocking thread * Wait until a valid template is available * Handle shutdown * Run mining on low priority threads * Ignore some invalid solutions * Use a difference nonce for each solver thread * Update TODOs * Change the patch into a renamed dependency to simplify crate releases * Clean up instrumentation and TODOs * Make RPC instances cloneable and clean up generics * Make LongPollId Copy so it's easier to use * Add API to restart mining if there's a new block template * Actually restart mining if there's a new block template * Tidy instrumentation * fixup! Move config validation into GetBlockTemplateRpcImpl::new() * fixup! Make RPC instances cloneable and clean up generics * Run the template generator and one miner concurrently * Reduce logging * Fix a bug in getblocktemplate RPC tip change detection * Work around some watch channel change bugs * Rate-limit template changes in the receiver * Run one mining solver per available core * Use updated C code with double-free protection * Update to the latest solver branch * Return and submit all valid solutions * Document what INPUT_LENGTH means * Fix watch channel change detection * Don't return early when a mining task fails * Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness * Make existing parallelism docs and configs consistent * Add a mining parallelism config * Use the minimum of the configured or available threads for mining * Ignore optional feature fields in tests * Downgrade some frequent logs to debug * Document new zebrad features and tasks * Describe the internal-miner feature in the CHANGELOG * Update dependency to de-duplicate equihash solutions * Use futures::StreamExt instead of TryStreamExt * Fix a panic message typo
2024-01-11 06:41:01 -08:00
miner_task_handle.abort();
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
// startup tasks
state_checkpoint_verify_handle.abort();
old_databases_task_handle.abort();
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
// Wait until the RPC server shuts down.
// This can take around 150 seconds.
//
// Without this shutdown, Zebra's RPC unit tests sometimes crashed with memory errors.
if let Some(rpc_server) = rpc_server {
info!("waiting for RPC server to shut down");
rpc_server.shutdown_blocking();
}
info!("exiting Zebra: all tasks have been asked to stop, waiting for remaining tasks to finish");
Download Zcash Sapling parameters and load them from cached files (#3057) * Replace Zcash parameters crates with pre-downloaded local parameter files * Download Zcash parameters using the `zcashd` script in CI and Docker * Add a zcash_proofs dependency to zebra-consensus * Download Sapling parameters using zcash_proofs, rather than fetch-params.sh * Add a new `zebrad download` subcommand This command isn't required for nomrmal usage. But it's useful when testing, or launching multiple Zebra instances. * Use `zebrad download` in CI to pre-download parameters * Log a helpful hint if downloading fails * Allow some duplicate dependencies currently hidden by orchard * Spawn a separate task to download Groth16 parameters * Run the parameter download with code coverage This avoids re-compining Zebra with and without coverage. * Update Cargo.lock after rebase * Try to pass `download` as an argument to `zebrad` in coverage CI * Fix copy and paste comment typos * Add path and download examples, like zcash_proofs * Download params in CI just like zcash_proofs does * Delete a redundant build step * Implement graceful shutdown for zebrad start * Send coverage summary to /dev/null when getting the params path * Use the correct parameters path and download commands in CI * Explain pre-downloads * Avoid calling params_folder twice * Rename parameter types and methods for consistency ```sh fastmod SaplingParams SaplingParameters zebra* fastmod Groth16Params Groth16Parameters zebra* fastmod PARAMS GROTH16_PARAMETERS zebra* fastmod params_folder directory zebra* ``` And a manual variable name tweak. * rustfmt * Remove a redundant coverage step Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-19 15:02:56 -08:00
exit_status
2020-06-15 17:07:55 -07:00
}
/// Returns the bound for the state service buffer,
/// based on the configurations of the services that use the state concurrently.
fn state_buffer_bound() -> usize {
fix(zebrad): accept default subcommand arguments and print consistent usage information for top-level 'help' subcommand (#6801) * updates Cargo.toml * Migrate to abscissa 0.7.0 * Avoid panic from calling color_eyre::install twice * Uses 'start' as the default subcommand * updates default cmd logic * Fixes minor cli issues * removes outdated check in acceptance test * Adds a test for process_cli_args, fixes version_args test. Adds -V to process_cli_args match case * Revert "fix(clippy): Silence future-incompat warnings until we upgrade Abscissa (#6024)" This reverts commit dd90f79b4824af6b1ae22ee69abdf0f605da84cd. * Drops the worker guard to flush logs when zebra shuts down * Adds cargo feature to clap * restores process_cli_args * updates deny.toml * Updates EntryPoint help template * Updates subcommand help msgs * removes trailing whitespace, capitalizes sentences * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * revert parts of revert "Revert fix(clippy): Silence future-incompat warnings until we upgrade Abscissa" * Applies suggestions from code review * Moves EntryPoint to its own module * fixes version_args test * Updates changelog * Prunes redundant test cases * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Revert "Prunes redundant test cases" This reverts commit 3f7397918489144805c17d0594775aa699e87b9d. * Update zebrad/src/commands/entry_point.rs Co-authored-by: teor <teor@riseup.net> * Add missing import * Updates `process_cli_args` to return a result --------- Co-authored-by: teor <teor@riseup.net>
2023-06-06 23:03:42 -07:00
let config = APPLICATION.config();
// Ignore the checkpoint verify limit, because it is very large.
//
// TODO: do we also need to account for concurrent use across services?
// we could multiply the maximum by 3/2, or add a fixed constant
[
config.sync.download_concurrency_limit,
config.sync.full_verify_concurrency_limit,
inbound::downloads::MAX_INBOUND_CONCURRENCY,
mempool::downloads::MAX_INBOUND_CONCURRENCY,
]
.into_iter()
.max()
.unwrap()
}
2020-06-15 17:07:55 -07:00
}
2020-06-16 11:02:01 -07:00
impl Runnable for StartCmd {
/// Start the application.
fn run(&self) {
info!("Starting zebrad");
fix(zebrad): accept default subcommand arguments and print consistent usage information for top-level 'help' subcommand (#6801) * updates Cargo.toml * Migrate to abscissa 0.7.0 * Avoid panic from calling color_eyre::install twice * Uses 'start' as the default subcommand * updates default cmd logic * Fixes minor cli issues * removes outdated check in acceptance test * Adds a test for process_cli_args, fixes version_args test. Adds -V to process_cli_args match case * Revert "fix(clippy): Silence future-incompat warnings until we upgrade Abscissa (#6024)" This reverts commit dd90f79b4824af6b1ae22ee69abdf0f605da84cd. * Drops the worker guard to flush logs when zebra shuts down * Adds cargo feature to clap * restores process_cli_args * updates deny.toml * Updates EntryPoint help template * Updates subcommand help msgs * removes trailing whitespace, capitalizes sentences * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * revert parts of revert "Revert fix(clippy): Silence future-incompat warnings until we upgrade Abscissa" * Applies suggestions from code review * Moves EntryPoint to its own module * fixes version_args test * Updates changelog * Prunes redundant test cases * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * Revert "Prunes redundant test cases" This reverts commit 3f7397918489144805c17d0594775aa699e87b9d. * Update zebrad/src/commands/entry_point.rs Co-authored-by: teor <teor@riseup.net> * Add missing import * Updates `process_cli_args` to return a result --------- Co-authored-by: teor <teor@riseup.net>
2023-06-06 23:03:42 -07:00
let rt = APPLICATION
.state()
.components_mut()
.get_downcast_mut::<TokioComponent>()
.expect("TokioComponent should be available")
.rt
.take();
rt.expect("runtime should not already be taken")
.run(self.start());
info!("stopping zebrad");
}
}
2020-06-16 11:02:01 -07:00
impl config::Override<ZebradConfig> for StartCmd {
// Process the given command line options, overriding settings from
// a configuration file using explicit flags taken from command-line
// arguments.
fn override_config(&self, mut config: ZebradConfig) -> Result<ZebradConfig, FrameworkError> {
if !self.filters.is_empty() {
config.tracing.filter = Some(self.filters.join(","));
}
Ok(config)
}
}