* updates comments
* adds check nullifier no dup fns for transactions
* Adds:
- check::anchors fn for tx iter
- TODO comments for unifying nullifiers and anchors checks
- new state request
Updates unknown anchor errors to accomodate tx-only check
Calls new state fn from transaction verifier
* updates check::anchors fns to use transactions
updates TransactionContextualValidity request to check sprout anchors
adds comment mentioning TransactionContextualValidity ignores UTXOs
* conditions new state req call on is_mempool
updates tests
* fix doc link / lint error
* checks for duplicate nullifiers with closures
* Update zebra-state/src/service/check/nullifier.rs
Co-authored-by: teor <teor@riseup.net>
* documents find_duplicate_nullifier params
moves if let statement into for loop
* renames new state req/res
* asserts correct response variant in tx verifier
* adds CheckBestChainTipShieldedSpends call in tx verifier to async checks
* re-adds tracing instrumentation to check::anchors fn
renames transaction_in_state to transaction_in_chain
* adds block/tx wrapper fns for anchors checks
* uses UnminedTx instead of transaction.hash()
deletes broken test
* updates new state req/res name
* updates tests and uses par_iter for anchors checks
* Updates check::anchors pub fn docs.
* Adds:
- comments / docs
- a TransactionError variant for ValidateContextError
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* moves downcast to From impl
rustfmt
* moves the ValidateContextError into an Arc
updates comments and naming
* leaves par_iter for another PR
* puts io::Error in an Arc
* updates anchors tests to call tx_anchors check
* updates tests to call tx_no_duplicates_in_chain
slightly improves formatting
* Update zebra-consensus/src/error.rs
Co-authored-by: teor <teor@riseup.net>
* moves Arc from HistoryError to ValidateContextError
Co-authored-by: teor <teor@riseup.net>
* Log zebrad config path as well as config values
* Remove duplicate config log
* Only log config path and data for server commands
* Use config ref instead
* print usage info for --help flag
* update known issues and help command's help text
* Update README.md
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* updates README
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Make the queue runner task shut down when the RpcImpl is dropped
* Move RPC server startup into the spawn() tokio future
* Return a shutdown handle from the RPC spawn() method
* Shut down the RPC server properly when Zebra shuts down
* Add a changelog entry for this security fix
* Call RpcServer::shutdown() when it is dropped, and wait
* Block on RPC server shutdown when Zebra's tasks have an error
* Add a getblocktemplate-rpcs feature to zebra-chain, and fix missing feature deps
* Add a coinbase transaction creation stub
* Add coinbase creation to zebra-chain
* Add coinbase creation and miner subsidy to zebra-consensus
* Add the miner config to the GetBlockTemplateRpcImpl
* Generate the coinbase transaction in the getblocktemplate RPC
* Provide fake valid block heights to getblocktemplate RPC tests
* Update getblocktemplate RPC snapshots
* Add a getblocktemplate.coinbase_tx deserialized transaction snapshot test
* Update snapshots
* Return funding stream outputs in the same order every time
* Update snapshots
* Fix a script bytes bug
* Update snapshots
* Use correct release for getblocktemplate config
* Include at least 2 full checkpoints in the lookahead limit
* Increase full sync timeout to 36 hours
* Only log "synced block height too far ahead of the tip" once
* Replace AboveLookaheadHeightLimit error with pausing the syncer
* Use AboveLookaheadHeightLimit for blocks a very long way from the tip
* Also add the getblocktemplate config, and fix the test message
* Remove an outdated TODO comment
* Allow syncing again when a small number of blocks are in the queue
* Allow some dead code
* Fix error text for state service for syncer
* Fix error handling in syncer
* cargo fmt --all
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* adds submitblock rpc method
* re-orders imports
* replaces thread::yield_now with async yield_now
* Fix doc warnings and unused variable warnings, add missing docs
* Mark work_id as optional
* Use the same ChainVerifier for downloaded and submitted blocks
* Revert unused changes & minor cleanups
* Document currently-unreachable code
* updates tests and submit_block response for AlreadyVerified error
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* changes names from BlockVerifier to ChainVerifier and block_verifier to chain_verifier to keep it consistent with naming in zebra-consensus
* move how to run the submit_block test example to acceptance.rs
* updates snapshot tests
* moved acceptance test to a separate file
* removes extra tower::ServiceBuilder::new(), updates docs
* updates vectors and snapshot tests, changes hex decoding error in submit_block method from server error to parse error
* hides errors module in zebra-rpc behind a feature flag and adds docs.
* Updates snapshot test, adds mod docs, moves HexData to its own mod, and removes the unrelated make_server_error refactor for now
* update submit block acceptance test mod doc
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* allow(clippy::result_large_err)
* Increase the async executor delay expected by tests
* Split getblocktemplate-rpcs OS tests into a separate matrix job
* Add new patch jobs
* allow(unknown_lints)
* Add a legacy_sigop_count field to VerifiedUnminedTx
* Add conversions from Vec<VerifiedUnminedTx> to block header roots
* Add fee and sigops field to block template transactions
* Fix up mempool request names
* Increase existing snapshot test coverage
* Document a new method parameter
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Add documentation for the getblocktemplate RPC
* Add a new mempool::Request::Transactions
* Add conversions from Vec<UnminedTx> to merkle::Root and AuthDataRoot
* Fill in the merkle root and auth data root fields
* Delete the Coinbase type, it's the same as Transaction
* Fill in some other existing types
* Add Hex serialization support to some zebra-chain types
* Add TransactionTemplate fields and fill some in
* Fix test hangs by spawning async tasks
* Add temporary workaround for no transactions in the block
* Encode hashes and roots as hex
* Update RPC snapshots
* Add a missing Request::Transactions handler
* Fix doc warnings
* Fix fee serialization
* Update snapshots for serialization changes
* Add a missing Cargo.lock change
* Change depends type
* Remove duplicate feature entry
* Document the new RPC feature
* Fix a comment typo
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Update default roots docs
* Fix comment typo
* Fix a comment typo
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Arya <aryasolhi@gmail.com>
* adds transactions to ChainTipBlock and rejects transactions from the mempool that have been invalidated by the latest block commit
* merges remove_same_effects in with reject_invalidated_transactions, moves nullifier retrieval to the new Storage method, rejects mined_ids, and updates tests
* updates DuplicateSpend's error message
* fixes tests
* Update zebrad/src/components/mempool/storage.rs
Co-authored-by: teor <teor@riseup.net>
* adds comment
* formatting for the proptest
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* waits for the chain tip update if setup adds mempool transactions and adds match arm for Response::Nil to MempoolTransactionIds request
* updates other match statements checking MempoolTransactionIds requests
* removes wait for chain tip update and waits for AdvertiseBlock earlier instead
* Add a new block commit task and channels, that don't do anything yet
* Add last_block_hash_sent to the state service, to avoid database accesses
* Update last_block_hash_sent regardless of commit errors
* Rename a field to StateService.max_queued_finalized_height
* Commit finalized blocks to the state in a separate task
* Check for panics in the block write task
* Wait for the block commit task in tests, and check for errors
* Always run a proptest that sleeps once
* Add extra debugging to state shutdowns
* Work around a RocksDB shutdown bug
* Close the finalized block channel when we're finished with it
* Only reset state queue once per error
* Update some TODOs
* Add a module doc comment
* Drop channels and check for closed channels in the block commit task
* Close state channels and tasks on drop
* Remove some duplicate fields across StateService and ReadStateService
* Try tweaking the shutdown steps
* Update and clarify some comments
* Clarify another comment
* Don't try to cancel RocksDB background work on drop
* Fix up some comments
* Remove some duplicate code
* Remove redundant workarounds for shutdown issues
* Remode a redundant channel close in the block commit task
* Remove a mistaken `!force` shutdown condition
* Remove duplicate force-shutdown code and explain it better
* Improve RPC error logging
* Wait for chain tip updates in the RPC tests
* Wait 2 seconds for chain tip updates before skipping them
* Remove an unnecessary block_in_place()
* Fix some test error messages that were changed by earlier fixes
* Expand some comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
* Actually drop children of failed blocks
* Explain why we drop descendants of failed blocks
* Clarify a comment
* Wait for chain tip updates in a failing test on macOS
* Clean duplicate finalized blocks when the non-finalized state activates
* Send an error when receiving a duplicate finalized block
* Update checkpoint block behaviour, document its consensus rule
* Wait for chain tip changes in inbound_block_height_lookahead_limit test
* Wait for the genesis block to commit in the fake peer set mempool tests
* Disable unreliable mempool verification check in the send transaction test
* Appease rustfmt
* Use clear_finalized_block_queue() everywhere that blocks are dropped
* Document how Finalized and NonFinalized clones are different
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* updates comments, renames send_process_queued, other minor cleanup
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* updates comments, renames send_process_queued, other minor cleanup
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* removes duplicate field definitions on StateService that were a result of a bad merge
* update NotReadyToBeCommitted error message
* Appear rustfmt
* Fix doc links
* Rename a function to initial_contextual_validity()
* Do error tasks on Err, and success tasks on Ok
* Simplify parent_error_map truncation
* Rewrite best_tip() to use tip()
* Rename latest_mem() to latest_non_finalized_state()
```sh
fastmod latest_mem latest_non_finalized_state zebra*
cargo fmt --all
```
* Simplify latest_non_finalized_state() using a new WatchReceiver API
* Expand some error messages
* Send the result after updating the channels, and document why
* wait for chain_tip_update before cancelling download in mempool_cancel_mined
* adds `sent_non_finalized_block_hashes` field to StateService
* adds batched sent_hash insertions and checks sent hashes in queue_and_commit_non_finalized before adding a block to the queue
* check that the `curr_buf` in SentHashes is not empty before pushing it to the `sent_bufs`
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix rustfmt
* Check for finalized block heights using zs_contains()
* adds known_utxos field to SentHashes
* updates comment on SentHashes.add method
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* return early when there's a duplicate hash in QueuedBlocks.queue instead of panicking
* Make finalized UTXOs near the final checkpoint available for full block verification
* Replace a checkpoint height literal with the actual config
* Update mainnet and testnet checkpoints - 7 October 2022
* Fix some state service init arguments
* Allow more lookahead in the downloader, but less lookahead in the syncer
* Add the latest config to the tests, and fix the latest config check
* Increase the number of finalized blocks checked for non-finalized block UTXO spends
* fix(log): reduce verbose logs for block commits (#5348)
* Remove some verbose block write channel logs
* Only warn about tracing endpoint if the address is actually set
* Use CloneError instead of formatting a non-cloneable error
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Increase block verify timeout
* Work around a known block timeout bug by using a shorter timeout
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a new block commit task and channels, that don't do anything yet
* Add last_block_hash_sent to the state service, to avoid database accesses
* Update last_block_hash_sent regardless of commit errors
* Rename a field to StateService.max_queued_finalized_height
* Commit finalized blocks to the state in a separate task
* Check for panics in the block write task
* Wait for the block commit task in tests, and check for errors
* Always run a proptest that sleeps once
* Add extra debugging to state shutdowns
* Work around a RocksDB shutdown bug
* Close the finalized block channel when we're finished with it
* Only reset state queue once per error
* Update some TODOs
* Add a module doc comment
* Drop channels and check for closed channels in the block commit task
* Close state channels and tasks on drop
* Remove some duplicate fields across StateService and ReadStateService
* Try tweaking the shutdown steps
* Update and clarify some comments
* Clarify another comment
* Don't try to cancel RocksDB background work on drop
* Fix up some comments
* Remove some duplicate code
* Remove redundant workarounds for shutdown issues
* Remode a redundant channel close in the block commit task
* Remove a mistaken `!force` shutdown condition
* Remove duplicate force-shutdown code and explain it better
* Improve RPC error logging
* Wait for chain tip updates in the RPC tests
* Wait 2 seconds for chain tip updates before skipping them
* Remove an unnecessary block_in_place()
* Fix some test error messages that were changed by earlier fixes
* Expand some comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
* Actually drop children of failed blocks
* Explain why we drop descendants of failed blocks
* Clarify a comment
* Wait for chain tip updates in a failing test on macOS
* Clean duplicate finalized blocks when the non-finalized state activates
* Send an error when receiving a duplicate finalized block
* Update checkpoint block behaviour, document its consensus rule
* Wait for chain tip changes in inbound_block_height_lookahead_limit test
* Wait for the genesis block to commit in the fake peer set mempool tests
* Disable unreliable mempool verification check in the send transaction test
* Appease rustfmt
* Use clear_finalized_block_queue() everywhere that blocks are dropped
* Document how Finalized and NonFinalized clones are different
* Use the same check as commit_finalized() for finalized block heights
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* use spawn_blocking to run zebra_state::init from start cmd
* uses zebra_state::spawn_init in copy-state command
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move version into a ConnectionInfo struct
* Add negotiated version to ConnectionInfo
Part of this change was generated using:
```
fastmod --fixed-strings ".version(" ".remote_version(" zebra-network
```
* Add the peer address to ConnectionInfo, add ConnectionInfo to Connection
* Return a Client instance from connect_isolated_* functions
This allows library users to access client ConnectionInfo.
* Add and improve debug formatting
* Add peer services and user agent to ConnectionInfo
* Export the Client type, and fix up a zebrad test
* Export types used by the public API
* Split VersionMessage into its own struct
* Use VersionMessage in ConnectionInfo
* Add a public API test for ConnectionInfo
* Wrap ConnectionInfo in an Arc
* Fix some doc links
* adds non-blocking writer for tracing subscriber
* use non_blocking writer for the fmt::Layer with the tokio-console feature as well
* adds doc comment to _guard field
* adds acceptance test
* update filter_handle type to use NonBlocking
* adds more detail on lossy non-blocking writer and sets tracing.filter to "trace" in acceptance test
* drops ZebradApp before process::exit(1) in the event of a FrameworkError
* reduces buffered lines limit to 8000
* adds tracing.buffer_limit config and some comments
* update acceptance.rs
* fix acceptance test
* fixes ambigious phrasing in comment
* updates zebrad/src/application.rs
* Find out what the join error is in the GitHub runner tests
* updates acceptance test to use recv_timeout instead of always waiting 10 seconds, removes unnecessary echo command, and reduces # of rpc requests to 500
* see if sleeping for a few seconds before exiting helps the macOS test pass
* Expand exit sleep docs
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* add grpc mempool test research
* add a config flag for mempool injection of transactions in test
* Only copy the inner state directory in the send transactions test
* Preload Zcash parameters in some transaction verification tests
* Add a block and transaction Hash method to convert from display order bytes
* Update test coverage docs
* Add debugging output for mempool transaction verification
* Test fetching sent mempool transactions using gRPC
* Add extra log checks to the send transaction test
* Wait for zebrad mempool activation before running gRPC tests
* Update send transaction test for lightwalletd not returning mempool transactions
* Check zebrad logs instead of disabled lightwalletd gRPCs
* Add a debug option that makes RPCs pretend the sync is finished
* Remove an unused debug option
* Remove unused test code and downgrade some logs
* Fix test log checks
* Fix some rustdoc warnings
* Fix a compilation error due to new function arguments
* Make zebrad sync timeouts consistent and remove outdated code
* Document how to increase temporary directory space for tests
* Stop checking for a log that doesn't always happen
* Remove some commented-out code
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Update a comment about run time
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Add new config to new tests from the `main` branch
* Add transactions to the list, rather than replacing the list with each new block
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Add a config for increasing the number of RPC threads
* Add unit tests for parallel RPC servers
* Refactor tests to remove duplicate code
* Update the README
* Actually use parallel threads in some RPC tests
* adds start as default subcommand for zebrad
* moves EntryPoint to submodule and adds a test
* moves all start tests to config_test to avoid listener conflicts
* Update zebrad/src/application/entry_point.rs docs
* Revert "moves all start tests to config_test to avoid listener conflicts"
This reverts commit 61ce46f5a1.
* Update based on test API changes from another PR
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* truncate the number of transactions in send transaction test
* Limit send transaction test to RPC queue length
Co-authored-by: teor <teor@riseup.net>
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography
* Improve anchor validation debugging and error messages
* Work around a test data bug, and save some CPU
* Remove redundant checks for empty shielded data
* Skip generating unused interstitial treestates
* Do disk fetches and quick checks, then CPU-heavy cryptography
* Wrap HistoryTree in an Arc in the state
* Run CPU-intensive chain validation and updates in parallel rayon threads
* Refactor to prepare for parallel tree root calculations
* Run finalized state note commitment tree root updates in parallel rayon threads
* Update finalized state note commitment trees using parallel rayon threads
* Fix a comment typo and add a TODO
* Split sprout treestate fetch into its own function
* Move parallel note commitment trees to zebra-chain
* Re-calculate the tree roots in the same parallel batches
* Do non-finalized note commitment tree updates in parallel threads
* Update comments about note commitment tree rebuilds
* Do post-fork tree updates in parallel threads
* Add a TODO for parallel tree updates in tests
* Fix broken intra-doc links
* Clarify documentation for sprout treestates
* Spawn large database reads into blocking tokio threads
* Concurrently read all blocks, headers, and transactions from disk
* Run zebra-state transaction deserialization on a rayon thread
* Update comments about note commitment tree rebuilds
* Add info-level logs to time note commitment tree rebuilds
* Log the number of rebuilt blocks after a fork, and the time per block
* Move humantime formats to zebra-chain
* Use human-friendly time formatting
* Only fetch block headers from the database to answer headers requests
* Move writing to the database to a blocking thread
* Add blocking threads to tests that need them
* Remove mempool downloader requirement for a populated state
And stop populating states that don't need it,
so we can use tokio::time::pause() in those tests.
* Improve debugging for an intermittent test failure
* Try to avoid a race condition populating the mempool in tests
* try to do deserialization of transaction in a rayon thread
* Try tokio::task::block_in_place instead
* fix tests
* add deserialize block into rayon pool
* fill some docs
Co-authored-by: teor <teor@riseup.net>
* Initialize the rayon threadpool with a new config for CPU-bound threads
* Verify proofs and signatures on the rayon thread pool
* Only spawn one concurrent batch per verifier, for now
* Allow tower-batch to queue multiple batches
* Fix up a potentially incorrect comment
* Rename some variables for concurrent batches
* Spawn multiple batches concurrently, without any limits
* Simplify batch worker loop using OptionFuture
* Clear pending batches once they finish
* Stop accepting new items when we're at the concurrent batch limit
* Fail queued requests on drop
* Move pending_items and the batch timer into the worker struct
* Add worker fields to batch trace logs
* Run docker tests on PR series
* During full verification, process 20 blocks concurrently
* Remove an outdated comment about yielding to other tasks
* Return the maximum checkpoint height from the chain verifier
* Return the verified block height from the sync downloader
* Track the verified height in the syncer
* Use a lower concurrency limit during full verification
* Get the tip from the state before the first verified block
* Limit the number of submitted download and verify blocks in a batch
* Adjust lookahead limits when transitioning to full verification
* Keep unused extra hashes and submit them to the downloader later
* Remove redundant verified_height and state_tip()
* Split the checkpoint and full verify concurrency configs
* Decrease full verification concurrency to 5 blocks
10 concurrent blocks causes 3 minute stalls on some blocks on my machine.
(And it has about 4x as many cores as a standard machine.)
* cargo +stable fmt --all
* Remove a log that's verbose with smaller lookahead limits
* Apply the full verify concurrency limit to the inbound service
* Add a summary of the config changes to the CHANGELOG
* Increase the default full verify concurrency limit to 30
* Document configs that need compile-time features
* Document features in zebrad/src/lib.rs
* Link to the feature list from the README
* Remove some outdated README info
* Move some detailed README info to the `zebrad` crate docs
* Improve time logging using humantime
* Only log full seconds, ignore the fractional part
* Move humantime_seconds to tracing::fmt
* Move the progress task to its own module
* Add missing humantime dependency
* Log the network upgrade in progress logs
* Log when Zebra verifies the final checkpoint
* Warn the user when Zebra cannot parse `zebrad.toml`
* Test that Zebra warns the user when it cannot parse `zebrad.toml`
* Fix up a mistaken merge change
* Suggest how to fix `zebrad.toml` when Zebra cannot parse it
Co-authored-by: teor <teor@riseup.net>
* change `initial_mainnet_peers` and `initial_testnet_peers` type to `IndexSet`
* add tests for zebra config files
* add serde feature to indexmap
* remove async
* update config
* fix `stored_config_path()`
* skip tests if config is not found
* improve error
* use CARGO_MANIFEST_DIR
* remove `stored_config_is_newest` test
* move `stored_config_works` test to the end of `valid_generated_config_test`
* space
* Decrease the default lookahead limit to 400
* Increase the block verification timeout to 10 minutes
* Halve the default concurrent downloads config
* Try to run the spawned download task before queueing the next download
* Allow verification to be cancelled if the verifier is busy
* delete old database directories
* check if state directory exists
* skip deleting when ephemeral
* split `check_and_delete_old_databases`
* move `check_and_delete_old_databases` to state
* spawn `check_and_delete_old_databases`
* simplity a bit
* fix(state): only delete old database directories inside the cache directory (#4631)
* Add function comments, tweak log
* Simplify version parsing
* Use spawn_blocking to launch the task on a separate thread, do the cleanup last
* Abort the cleanup task when Zebra exits
* Split directory deletion into its own function, handle ownership
* Rename cache_dir to state_dir
* If an outdated state directory is outside the cache directory, don't delete it
* Minimise diffs
* add test
* fix typos
Co-authored-by: teor <teor@riseup.net>
* add `canonicalize` to test regex
* add another match to test
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Disable the flamegraph feature by default at compile time
* Disable the journald feature by default at compile time
* Also disable inferno dependency, and rearrange features
* Disable the prometheus feature by default at compile time
* Disable the tracing filter reload feature by default at compile time
* Disable tests when corresponding features are disabled
* Add compile-time tracing features to user docs
* Add compile-time features to the metrics user docs
* Document diagnostics as part of the start command tasks and services
* breaking(diagnostics): rename "enable-sentry" feature to "sentry" (#4623)
* Also skip conflict tests when those ports are disabled
* breaking(diagnostics): rename "enable-sentry" feature to "sentry"
This is mostly:
```sh
fastmod enable-sentry sentry
```
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Always activate tokio/tracing feature
And always build tests with all tokio features.
* Refactor tracing-subscriber init to simplify it
* Add the tokio-console feature and dependencies
* Add optional tokio-console support, and log the installed tracing layers at info level
Uses a tracing Registry for tokio-console, and a fmt::Subscriber otherwise.
* Add some TODOs based on tracing-subscriber features
* Fix up some spans
* Add a TODO for fixing a log filter bug in tokio-console mode
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fa.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
* Fix some warnings produced by `cargo doc`
* Fix some rustdoc warnings
* Fix some warnings
* Refactor some changes
* Fix some rustdoc warnings
* Fix some rustdoc warnings
* Resolve various TODOs
Co-authored-by: teor <teor@riseup.net>
* Fix some unresolved links
* Allow links to private items
* Fix some unresolved links
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* use `humantime_serde` for config durations
* move debug config option to the bottom
* fix deserialization
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move peer address validation into its own module
* Add a network parameter to AddressBook and some MetaAddr methods
* Reject invalid initial peers, and connect to them in preferred order
* Reject Flux/ZelCash and misconfigured Zcash peers
* Prefer canonical Zcash ports
* Make peer_preference into a struct method
* Prefer peer addresses with canonical ports for outbound connections
* Also ignore the Zcash regtest port
* Document where field and variant order is required for correctness
* Use the correct peer count
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fa.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
* Fix some warnings produced by `cargo doc`
* Fix some rustdoc warnings
* Fix some warnings
* Refactor some changes
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Upgrade tracing and related dependencies
```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber
color-eyre
tracing-flame
tracing-journald
sentry
sentry-tracing
metrics
metrics-exporter-prometheus
reqwest
```
* Update duplicate dependency checks
* Enable the tracing/env-filter feature
* Fix type inference for metrics
Manual changes, plus:
```sh
fastmod "as _" "as f64"
```
* Tidy up some unrelated test code
* Update metrics-exporter-prometheus API
And make unused dependencies optional.
* Adjust test regexes to new tracing format
Also fix some regex bugs, and refactor to simplify.
* Disable color-eyre span traces and track caller in release builds
* Add a feature that enables extra debugging in release builds
* Clean up some redundant features
* Increase a test timeout
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fa.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Move `init_tracing` to `lib.rs`
Allow it to be reused by other binaries.
* Fix `hex` dependency inclusion in `zebra-chain`
It would previously fail because `hex` was being included without the
`serde` feature, even though we required that for `transparent::Script`.
* Implement `FromStr` for `Network`
Make it easy to receive it as a command-line parameter.
* Add `zebra-tip-height` utility
Obtains the chain tip height of a directory containing Zebra state.
* Remove Tokio dependency from `zebra-utils`
It wasn't actually used by the `zebra-tip-height` utility.
* Remove `BoxStateService` type alias
It's not needed if we don't return the unused state service. This also
allows removing the `tower` dependency.
* Remove unnecessary attribute
Leftover from copied code.
* Make `cache_dir` argument optional
Fallback to the default Zebra state cache directory.
* Remove added newline
Minor formatting fix, to avoid adding an unnecessary newline.
* Move `tip-height` command into `zebrad`
Make it available as a sub-command in `zebrad`.
* Make some zebrad sub-commands only log warnings
Co-authored-by: teor <teor@riseup.net>
* Increase UTXO lookup timeout
Avoid block validation failures because UTXOs aren't available on time.
* Increase the block verification timeout
Attempt to reduce the synchronization restarts and consequently improve
performance.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Return BlockDownloadVerifyError from download_and_verify
* Check block requests and genesis for temporary errors
* Ignore DuplicateBlockQueuedForDownload as a temporary error
* Propagate error info to the syncer main loop
* Sleep after temporary genesis download and verify errors
* Increase the block download retry limit to 3
* Remove an obsolete timing check from the tests
We cancel all block downloads for each sync restart,
so we don't need to wait for the maximum hedged download time.
* Add a rpc queue
* Implement the rpc queue
* Add rpc queue tests
* Remove mutex, use broadcast channel
* Have order and limit in the queue
* fix multiple transactions channel
* Use a network argument
* Use chain tip to calculate block spacing
* Add extra time
* Finalize the state check test
* Add a retry test
* Fix description
* fix some docs
* add additional empty check to `Runner::run`
* remove non used method
* ignore some errors
* fix some docs
* add a panic checker to the queue
* add missing file changes for panic checker
* skip checks and retries if height has not changed
* change constants
* reduce the number of queue test cases
* remove suggestion
* change best tip check
* fix(rpc): Check for panics in the transaction queue (#4046)
* Check for panics in the RPC transaction queue
* Add missing pin! and abort in the start task
* Check for transaction queue panics in tests
* Fixup a new RPC test from the main branch
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Repoint zebra image links to our new zfnd.org site for now
* Remove images/
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
- use info level, there is nothing the user needs to do,
particularly for a single error
- explain that the errors are temporary
- hide backtraces, because they look like crashes
* Refactor to split `ChainSync::sync` method in two
Replace the use of loop labels and `continue` for control flow, and use
early return from a separate method instead. This also allows removing
the `started_once` flag.
* Refactor to create `handle_block_response` helper
Reduce duplicate code and make the main synchronization methods a little
more concise to improve readability.
* Only cancel downloads in case of error
Leave active downloads running if the tips have been exhausted, because
it could have reached the chain tip.
* Add the ChainTip and Network to RpcImpl
* Add the new RpcImpl fields to the tests
* Simplify RPC version field using generics
* Temporarily allow unused struct fields
* Use the read-only state service for RPCs
* Refactor non-finalized block lookup into Chain
* Implement the read-only state block request
* Drop the Chain watch channel lock before accessing the finalized state
* Add state service module docs and cleanup
* Move and add finalized state methods
* Add chain and non-finalized state methods
* Cleanup methods and imports
* Create a ReadStateService type
* Add a stub service implementation
* Add a TODO
* Update ReadStateService request stubs with RPC names and tickets
* Documentation updates
* Make RPC State generic bounds accept a buffered state and a read-only state
* Doc updates
* Add missing proptest-impl feature in RPC dev dependencies
* feature(rpc): start adding a `getblock` method
* fix(rpc): replace oneshot
* fix(rpc): replace a panic with error
* fix(rpc): fix test
* feature(rpc): add hex to response
* refactor(rpc): use generic instead of alias
* docs(rpc): improve docs for getblock method
* test(rpc): add a test for getblock method
* deps(rpc): remove non needed tower features
Co-authored-by: teor <teor@riseup.net>
* docs(rpc): add a note to getblock doc
* refactor(rpc): replace alias
* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format
* tests(rpc): add network argument to `populated_state()`
* refactor(rpc): use an error for state service readiness
* fix(rpc): add parameter
* fix(rpc): clippy
* nit(rpc): remove new line from imports
* fix(rpc): remove commented code
* fix(rpc): simplify error
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use a `SerializedBlock` type to help serializing blocks (#3725)
* Create a `SerializedBlock` helper type
Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.
* Use `into_iter` instead of `iter`
There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.
* Use `SerializedBlock` in `GetBlock`
Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.
The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.
* feature(rpc): add suggestions from code review
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* tests(rpc): make sure mempool has no requests in get_block test
* fix(rpc): change height argument type in getblock method
* fix(rpc): rustfmt
* fix(rpc): replace panic
* fix(rpc): change getblock response
* fix(rpc): fix lightwalletd test
* tests(rpc): add a getblock error test
* fix(rpc): try another regex
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Stub `sendrawtransaction` RPC method
Register the RPC method, and stub an implementation that currently just
panics. The method has a single `String` parameter with the hexadecimal
string of the raw transaction's bytes and returns a
`SentTransactionHash` wrapper type that's just a hexadecimal `String` of
the sent transaction's hash.
* Add mempool service instance to `RpcImpl`
Use a type parameter to represent the mempool service using the
interface defined by `zebra-node-services`.
* Update test vector to use a mock mempool service
Update the test to be compatible with the changes to `RpcImpl`. The mock
mempool service is expected to not be used during the test.
* Use a `tower::Buffer` for the mempool service
Make it simpler to send requests to the service in a concurrent manner.
* Return a `Future` from `send_raw_transaction`
Make the call asynchronous.
* Implement `sendrawtransaction` RPC
Deserialize the transaction and send it to be queued for verification
and subsequent inclusion in the mempool.
* Test if mempool receives sent raw transaction
Use a mock service as the mempool service and check that it receives a
sent raw transaction.
* Test using non-hexadecimal string parameter
The method should return an error.
* Test with bytes that fail deserialization
Check that the method returns an invalid parameters error if the input
can't be deserialized as a `Transaction`.
* Test if mempool errors are forwarded to caller
Mempool service errors should be sent back to the remote caller as
server errors.
* Test transactions rejected by the mempool service
Transactions that are rejected by the mempool service should result in
a server error being sent to the caller.
* Improve error message
Add the word "structurally" to make it clear that the issue is in the
transaction's deserialization.
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Add note regarding missing `allowhighfees` param.
The parameter isn't supported yet because `lightwalletd` doesn't use it.
* Update the documentation to be consistent
Follow the convention adopted by the `get_info` RPC method.
* Implement `ToHex` and `FromHex` for `Hash`
Make it easier to generate hexadecimal strings from `transaction::Hash`
instances.
* Use `ToHex` in `Debug` and `Display`
Reduce repeated code.
* Refactor to add `bytes_in_display_order` method
Use it to remove repeated code and improve clarity a bit.
* Use `hex::serialize` to serialize transaction hash
Make the type stricter in its contents, while still serializing the
transaction has as a hexadecimal string.
* Simplify serialization attribute
Deserialization should also use `hex::deserialize`, so using the shorter
attribute makes things easier to read and more future proof.
* Update zebra-chain/src/transaction/hash.rs
* Remove unnecessary lifetime
The anonymous lifetime is automatically inferred by the compiler.
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Stub `sendrawtransaction` RPC method
Register the RPC method, and stub an implementation that currently just
panics. The method has a single `String` parameter with the hexadecimal
string of the raw transaction's bytes and returns a
`SentTransactionHash` wrapper type that's just a hexadecimal `String` of
the sent transaction's hash.
* Add mempool service instance to `RpcImpl`
Use a type parameter to represent the mempool service using the
interface defined by `zebra-node-services`.
* Update test vector to use a mock mempool service
Update the test to be compatible with the changes to `RpcImpl`. The mock
mempool service is expected to not be used during the test.
* Use a `tower::Buffer` for the mempool service
Make it simpler to send requests to the service in a concurrent manner.
* Return a `Future` from `send_raw_transaction`
Make the call asynchronous.
* Implement `sendrawtransaction` RPC
Deserialize the transaction and send it to be queued for verification
and subsequent inclusion in the mempool.
* Test if mempool receives sent raw transaction
Use a mock service as the mempool service and check that it receives a
sent raw transaction.
* Test using non-hexadecimal string parameter
The method should return an error.
* Test with bytes that fail deserialization
Check that the method returns an invalid parameters error if the input
can't be deserialized as a `Transaction`.
* Test if mempool errors are forwarded to caller
Mempool service errors should be sent back to the remote caller as
server errors.
* Test transactions rejected by the mempool service
Transactions that are rejected by the mempool service should result in
a server error being sent to the caller.
* Improve error message
Add the word "structurally" to make it clear that the issue is in the
transaction's deserialization.
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Add note regarding missing `allowhighfees` param.
The parameter isn't supported yet because `lightwalletd` doesn't use it.
* Update the documentation to be consistent
Follow the convention adopted by the `get_info` RPC method.
* Remove mempool service usage line
It contained incomplete information that's not really necessary.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Fix formatting
`rustfmt` was not executed on the file for the previous commit because I
had edited it on GitHub.
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* tests(rpc): add an rpc endpoint test
* tests(rpc): add an rpc port conflict test
* tests(rpc): refactor some imports
* tests(rpc): fix failures, make test more complete
* tests(rpc): parse json response for better coverage
* tests(rpc): change request
* tests(rpc): wait until port is open in rpc_endpoint test
* tests(rpc): add a delay between launching 2 nodes
* tests(rpc): try 5 seconds
* refactor(rpc): open rpc server faster
* tests(rpc): extend `LAUNCH_DELAY` to 15 seconds
* fix(rpc): disable rpc_conflict test for windows
* fix(ci): skip the RPC tests if the network is disabled
* rustfmt
* fix(zebrad/test): test function return type
* tests(rpc): print server output in assert
* fix(rpc): fix acceptance test looking for string in `build` field
* fix(rpc): reduce the number of acceptable characters in version output
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feature(rpc): add getinfo subversion field and getinfo docs
* feature(rpc): add getinfo build field
* refactor(rpc): replace the lazy_static
* docs(rpc): fic typo, add link to zcashd ticket
* tests(rpc): add getinfo unit test
* docs(rpc): complete comment
* Create new empty `zebra-node-services` crate
The goal is to store the mempool `Request` and `Response` types so that
the `zebra-rpc` crate can interface with the mempool service without
having to import `zebrad`.
* Move `Gossip` mempool type into new crate
It is required by the `Request` type, which will be moved next.
* Add documentation to `Gossip` variants
Avoid having to add an exception to allow undocumented code.
* Move `mempool::Request` type to new crate
The first part of the service interface definition. Usages have been
changed to refer to the new crate directly, and since this refactor is
still incomplete, some `mp` aliases are used in a few places to refer to
the old module.
* Create an `UnboxMempoolError` helper trait
Centralize some common code to extract and downcast boxed mempool
errors. The `mempool::Response` will need to contain a `BoxError`
instead of a `MempoolError` when it is moved to the
`zebra-node-services` crate, so this prepares the tests to be updated
with less changes.
* Use `UnboxMempoolError` in tests
Make the necessary changes so that the tests are ready to support a
`BoxError` in the `mempool::Response` type.
* Use `BoxError` in `mempool::Response`
Prepare it to be moved to the `zebra-node-services` crate.
* Move `mempool::Response` to `zebra-node-services`
Update usages to import from the new crate directly.
* Remove `mp` aliases for mempool component module
Use any internal types directly instead.
* Replace `tower::BoxService` with custom alias
Remove the dependency of `zebra-node-services` on `tower`.
* Move `Gossip` into a separate `sub-module`
Keep only the main `Request` and `Response` types in the `mempool`
module.
* Use `crate::BoxError` instead of `tower::BoxError`
Follow the existing convention.
* Add missing `gossip.rs` module file
It was missing from a previous refactor commit.
* feature(rpc): add an rpc component
* feat(rpc): add a stub for getblockchaininfo
This is the first RPC used by lightwalletd, so we need it for testing.
* fix(rpc): remove non-standard "jsonrpc: 1.0" from lightwalletd
* fix(rpc): re-enable default RPC security checks
* deps(rpc): remove not needed dependency
* fix(rpc): check if RPC task has stopped
* fix(rpc): reduce config by using Option
* fix(rpc): use tokio executor
* security(rpc): turn off rpc by default
* docs(rpc): update a TODO comment
Co-authored-by: teor <teor@riseup.net>
* fix(rpc): blocking tasks
Co-authored-by: teor <teor@riseup.net>
* rename(rpc): rpc.rs to methods.rs
* refactor(rpc): move the server to the zebra-rpc crate
* fix(rpc): clippy derive Default for RPC Config
* fix(dependencies): remove unused dependency features in zebra-rpc
We expect to use all the listed tokio features
to implement and test RPC methods.
* doc(rpc): fix testnet port, add security note
* fix(rpc): change Rust function names and update method doc TODOs
* fix(rpc): add "TODO" to fake RPC responses
* doc(rpc): update module docs
* fix(rpc): simplify server struct derives
* fix(rpc): simplify server code
* doc(rpc): explain how request fixes securely handle user-supplied data
* refactor(rpc): move the compatibility fix to a separate module
* fix(rpc): move the open log inside the spawn, and instrument it
* doc(rpc): fix toml format and provide a config example
Co-authored-by: teor <teor@riseup.net>
* feat(log): log current height when logging sync progress
* fix(test): log the specific error when full sync tests fail
* doc(start): remove an obsolete TODO
We can't decrease this log level, because the tests rely on it.
* fix(test): wait until mempool activates in full sync tests
Changes the log message and log test so that the test only finishes
when the mempool has activated.
There is still a race condition here, between the log timer and
mempool activation. But it should be very rare, because the mempool
is activated immediately when `is_close_to_tip()` becomes true.
* fix(test): warn when Zebra stalls below the maximum checkpoint height
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* feat(log): warn when sync stalls downloading the genesis block
* fix(test): warn when the state hasn't committed a block for a long time
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* doc(test): update some sync acceptance test comments
* fix(log): use Display formatting to log chrono::Duration
Debug formatting is complicated and hard to read.
* fix(log): stop saying that we've activated the mempool without checking it
We're not checking if the mempool is active, so we can't say that.
* fix(log): minor tidying and TODOs
* fix(doc): fix a typo in the tests
* fix(log): explain the post-checkpoint blocks in progress warning calculations
* fix(doc): explain what could happen if we don't wait for extra blocks
* fix(log): add a percent symbol to a percent log
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Allow forcing colored output in `zebrad`
Add a configuration item that allows forcing Zebra to output in color
mode even if the output device is not a terminal.
* Allow enabling colored output from Zebra in tests
Force Zebrad instances to use colored output if the
`ZEBRA_FORCE_USE_COLOR` environment variable is set.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(network): split synthetic NotFoundRegistry from message NotFoundResponse
* docs(network): Improve `notfound` message documentation
* refactor(network): Rename MustUseOneshotSender to MustUseClientResponseSender
```
fastmod MustUseOneshotSender MustUseClientResponseSender zebra*
```
* docs(network): fix a comment typo
* refactor(network): remove generics from MustUseClientResponseSender
* refactor(network): add an inventory collector to Client, but don't use it yet
* feat(network): register missing peer responses as missing inventory
We register this missing inventory based on peer responses,
or connection errors or timeouts.
Inbound message inventory tracking requires peers to send `notfound` messages.
But `zcashd` skips `notfound` for blocks, so we can't rely on peer messages.
This missing inventory tracking works regardless of peer `notfound` messages.
* refactor(network): rename ResponseStatus to InventoryResponse
```sh
fastmod ResponseStatus InventoryResponse zebra*
```
* refactor(network): rename InventoryStatus::inner() to to_inner()
* fix(network): remove a redundant runtime.enter() in a test
* doc(network): the exact time used to filter outbound peers doesn't matter
* fix(network): handle block requests slightly more efficiently
* doc(network): fix a typo
* fmt(network): `cargo fmt` after rename ResponseStatus to InventoryResponse
* doc(test): clarify some test comments
* test(network): test synthetic notfound from connection errors and peer inventory routing
* test(network): improve inbound test diagnostics
* feat(network): add a proptest-impl feature to zebra-network
* feat(network): add a test-only connect_isolated_with_inbound function
* test(network): allow a response on the isolated peer test connection
* test(network): fix failures in test synthetic notfound
* test(network): Simplify SharedPeerError test assertions
* test(network): test synthetic notfound from partially successful requests
* test(network): MissingInventoryCollector ignores local NotFoundRegistry errors
* fix(network): decrease the inventory rotation interval
This stops us waiting 3-4 sync resets (4 minutes) before we retry a missing block.
Now we wait 1-2 sync resets (2 minutes), which is still a reasonable rate limit.
This should speed up syncing near the tip, and on testnet.
* fmt(network): cargo fmt --all
* cleanup(network): remove unnecessary allow(dead_code)
* cleanup(network): stop importing the whole sync module into tests
* doc(network): clarify syncer inventory retry constraint
* doc(network): add a TODO for a fix to ensure API behaviour remains consistent
* doc(network): fix a function doc typo
* doc(network): clarify how we handle peers that don't send `notfound`
* docs(network): clarify a test comment
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* refactor(network): rename Advertised to Available
```sh
fastmod Advertised Available zebra*
fastmod advertised available zebra*
```
* refactor(network): allow different available and missing types inside an InventoryStatus
And rename it to ResponseStatus.
Split the methods between ResponseStatus and an InventoryStatus alias.
* refactor(network): add a block_hash convenience method to InventoryHash
* test(network): improve failure logs for connection tests
* fix(inbound): move address sanitization into the response future
* feat(network): send notfound when Zebra doesn't have a block or transaction
* doc(network): move module docs to the top of each module
This makes them more likely to get updated when the module changes.
* fix(network): stop sending unsupported missing inventory types to the registry
* test(network): inbound messages are forwarded to the registry
* test(inbound): test Peers requests to the inbound service, directly and via TCP
* test(network): notfound block responses are sent by the inbound service
* test(network): notfound tx responses are sent by the inbound service
* test(network): increase sync test mock service timeout
The code that these tests use hasn't actually changed much,
and they are only failing on some platforms (coverage, macOS).
So it seems like the extra concurrent inbound tests have pushed them
past their time limit.
(Perhaps due to TCP system calls, or extra serialization work.)
* doc(network): fix typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* test(network): remove unnecessary multi-threaded runtime from tests
This prevents `MockService<zebra_state>` timeouts
in the `sync_block_too_high_extend_tips` test,
at the cost of reducing coverage of different execution orders.
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Log chain progress while Zebra is syncing
This helps test if the chain tip estimate is accurate,
and helps diagnose problems during full sync tests.
* Update to the latest chain tip estimate API
* Remove redundant documentation
The documentation was exactly the same as the documentation from the
trait.
* Calculate a mock time block delta for tests
Simulate a block being added to the chain with a random block time based
on the previous block time and the target spacing time.
* Add a `time` field to `ChainTipBlock`
Store the block time so that it's ready for a future chain that allows
obtaining the chain tip's block time.
* Add `ChainTip::best_tip_block_time` method
Allow obtaining the bes chain tip's block time.
* Add method to obtain both height and block time
Prevent any data races by returning both values so that they refer to
the same chain tip.
* Add `NetworkUpgrade::all_target_spacings` method
Returns all the target spacings defined for a network.
* Create a `NetworkChainTipEstimator` helper type
Isolate the code to calculate the height estimation in a new type, so
that it's easier to understand and doesn't decrease the readability of
the `chain_tip.rs` file.
* Add `ChainTip::estimate_network_chain_tip_height`
This is more of an extension method than a trait method. It uses the
`NetworkChainTipHeightEstimator` to actually perform the estimation, but
obtains the initial information from the current best chain tip.
* Fix typo in documentation
There was an extra closing bracket in the summary line.
* Refactor `MockChainTipSender` into a separate type
Prepare to allow mocking the block time of the best tip as well as the
block height.
* Allow sending mock best tip block times
Add a separate `watch` channel to send the best tip block times from a
`MockChainTipSender` to a `MockChainTip`.
The `best_tip_height_and_block_time` implementation will only return a
value if there's a height and a block time value for the best tip.
* Fix off-by-one height estimation error
Use Euclidean division to force the division result to round down
instead of rounding towards zero. This fixes an off-by-one error when
estimating a height that is lower than the current height, because the
fractionary result was being discarded, and it should have forced the
height to go one block back.
* Fix panics on local times very far in the past
Detect situations that might cause the block height estimate to
underflow, and return the genesis height instead.
* Fix another off-by-one height estimation error
The implementation of `chrono::Duration::num_seconds` adds one to the
number of seconds if it's negative. This breaks the division
calculation, so it has to be compensated for.
* Test network chain tip height estimation
Generate pairs of block heights and check that it's possible to estimate
the larger height from the smaller height and a displaced time
difference.
* Use string literal to format unreachable message
* Try removing the redundant format
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(state): set state concurrency based on other services' concurrency
* fix(sync): increase the sync downloader lookahead limit
It seems like the recent tokio upgrade made this code even more efficient,
so on testnet we can have around 6000 blocks in flight.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feat(log): log the state tip height as part of sync progress logs
* fix(log): downgrade some verbose state logs to debug
* feat(log): log successful gossiped block verification at info level
These logs help us diagnose slow progress near the tip.
There won't be very many of these logs,
because they only happen near the tip.
* fix(log): spawn top-level tasks within the global Zebra tracing span
* fix(log): spawn blocking top-level tasks within the global Zebra tracing span
Co-authored-by: teor <teor@riseup.net>
* cancel background database tasks in `FinalizedState` destructor
* use `shutdown_timeout()`
* Log info-level messages while waiting for background tasks to shut down
* Cancel background tasks during debug_stop_at_height shutdown
This commit moves the database shutdown code into a common function.
* Create a constant for the tokio timeout
* Add a test script for Zebra shutdown errors
* Increase the shutdown timeout to 20 seconds for slower machines
* add title to building zebra
* use imported duration
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move standard lints into .cargo/config.toml
* Ignore "wrong self convention" in a futures-based trait
This lint might only trigger on beta or nightly at the moment.
* Warn if future incompatibile code is added to Zebra
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Refactor so that RetryLimit::Future is std::marker::Sync
* Make the syncer future std::marker::Send by spawning tips futures
* Download synced blocks in chain order, not HashSet order
* Improve MockService failure messages
* Add closure-based responses to the MockService API
* Move MockChainTip to zebra-chain
* Add a MockChainTipSender type alias
* Support MockChainTip in ChainSync and its downloader
* Add syncer tests for obtain tips, extend tips, and wrong block hashes
* Add block too high tests for obtain tips and extend tips
* Add syncer tests for duplicate FindBlocks response hashes
* Allow longer request delays for mocked services in syncer tests
* Add a copy-state command, which copies blocks between two state services
* Check blocks were written correctly
* Add extra logging to debug shutdown
* Add a block height limit argument
* Let the target state start from any height
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Move `MockedClientHandle` to `peer` module
It's more closely related to a `Client` than the `PeerSet`, and this
prepares it to be used by other tests.
* Rename `MockedClientHandle` to `ClientTestHarness`
Reduce confusion, and clarify that the client is not mocked.
Co-authored-by: teor <teor@riseup.net>
* Add clarification to `mock_peers` documentation
Explicitly say how the generated data is returned.
* Rename method to `wants_connection_heartbeats`
The `Client` service only represents one direction of a connection, so
`is_connected` is not the exact term.
Co-authored-by: teor <teor@riseup.net>
* Mock `Client` instead of `LoadTrackedClient`
Move where the conversion from mocked `Client` to mocked
`LoadTrackedClient` in order to make the test helper more easily used by
other tests.
* Use `ClientTestHarness` in `initialize` tests
Replace the boilerplate code to create a fake `Client` instance with
usages of the `ClientTestHarness` constructor.
* Allow receiving requests from `Client` instance
Create a helper type to wrap the result, to make it easier to assert on
specific events after trying to receive a request.
* Allow inspecting the current error in the slot
Share the `ErrorSlot` between the `Client` and the handle, so that the
handle can be used to inspect the contents of the `ErrorSlot`.
* Allow placing an error into the `ErrorSlot`
Assuming it is initially empty. If it already has an error, the code
will panic.
* Allow gracefully closing the request receiver
Close the endpoint with the appropriate call to the `close()` method.
* Allow dropping the request receiver endpoint
Forcefully closes the endpoint.
* Rename field to `client_request_receiver`
Also rename the related methods to include
`outbound_client_request_receiver` to make it more precise.
Co-authored-by: teor <teor@riseup.net>
* Allow dropping the heartbeat shutdown receiver
Allows the `Client` to detect that the channel has been closed.
* Rename fn. to `drop_heartbeat_shutdown_receiver`
Make it clear that it affects the heartbeat task.
Co-authored-by: teor <teor@riseup.net>
* Move `NowOrLater` into a new `now-or-later` crate
Make it easily accessible to other crates.
* Add `IsReady` extension trait for `Service`
Simplifies checking if a service is immediately ready to be called.
* Add extension method to check for readiness error
Checks if the `Service` isn't immediately ready because a call to
`ready` immediately returns an error.
* Rename method to `is_failed`
Avoid negated method names.
Co-authored-by: teor <teor@riseup.net>
* Add a `IsReady::is_pending` extension method
Checks if a `Service` is not ready to be called.
* Use `ClientTestHarness` in `Client` test vectors
Reduce repeated code and try to improve readability.
* Create a new `ClientTestHarnessBuilder` type
A builder to create test `Client` instances using mock data which can be
tracked and manipulated through a `ClientTestHarness`.
* Allow configuring the `Client`'s mocked version
Add a `with_version` builder method.
* Use `ClientTestHarnessBuilder` in `PeerVersions`
Use the builder to set the peer version, so that the `version` parameter
can be removed from the constructor later.
* Use a default mock version where possible
Reduce noise when setting up the harness for tests that don't really
care about the remote peer version.
* Remove `Version` parameter from the `build` method
The `with_version` builder method should be used instead.
* Fix some typos and outdated info in the release checklist
* Add extra client tests for zero and multiple readiness checks (#3273)
And document existing tests.
* Replace `NowOrLater` with `futures::poll!` (#3272)
* Replace NowOrLater with the futures::poll! macro in zebrad
* Replace NowOrLater with the futures::poll! macro in zebra-test
* Remove the now-or-later crate
* remove unused imports
* rustfmt
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Tweak crawler timings so peers are more likely to be available
* Tweak min peer connection interval so we try all peers
* Let other tasks run between fanouts, so we're more likely to choose different peers
* Let other tasks run between retries, so we're more likely to choose different peers
* Let other tasks run after peer crawler DemandDrop
This makes it more likely that peers will become ready.
* Document the chain verifier
* Drop gossiped blocks that are too far ahead of the tip
* Add extra gossiped block metrics
* Allow extra gossiped blocks, now we have a stricter limit
* Fix a comment
* Check the exact number of blocks in a downloaded block response
* Drop synced blocks that are too far ahead of the tip
* Add extra synced block metrics
* Test dropping gossiped blocks that are too far ahead of the tip
* Allow an extra checkpoint's worth of blocks in the verifier queues
* Actually let's try two extra checkpoints
* Scale extra height limit with lookahead limit
* Also drop blocks that are behind the finalized tip
* Downgrade a noisy log
* Use a debug log for already verified gossiped blocks
* Use debug logs for already verified synced blocks
* Use a named CancelHeartbeatTask unit struct for the channel type
* Prefer cancel handles in selects, if both are ready
* Fix message metrics to just show the command name
* Add metrics for internal requests and responses
* Add internal requests and responses to the messages dashboard
* Add a canceled metric, and peer addresses to request and response metrics
* Add a canceled messages graph
* Add connection state metrics for currently open connections
* Fix the connection state graph with new metrics
* Always send an error before dropping pending responses
* Move error detail logging into `fail_with`
* Delete an unused timer future
* Make error strings in metrics less verbose
* Downgrade some error logs to info
* Remove a redundant expect
* Avoid unnecessary allocations for connection state metrics
* Fix missed updates to mempool and block gossip metrics
* Stop useless crawler attempts when there are no peers and no crawl responses
* Disable GitHub bug report URLs when the disk is full
* Add help text for the `zebrad start` tracing filter option
* Use a single-thread shared Tokio runtime
This allows it to pause the time and more closely resembles the
environment that's set by default for asynchronous tests.
* Add a `zebra_test::init_async` helper function
Calls `zebra_test::init` but also constructs a single-thread Tokio
runtime and returns it. This makes it simpler to initialize asynchronous
tests that can't use the `#[tokio::test]` attribute.
* Replace usages of `Runtime::new` in tests
Use the new `zebra_test::init_async()` helper function instead.
* Replace `runtime::Builder::new_current_thread()`
Use the new `zebra_test::init_async()` helper function instead.
* Replace `runtime::Builder::new_multi_thread()`
Use the new `zebra_test::init_async()` helper function instead. The test
with the change doesn't necessarily have to use a multi-thread runtime.
* Refactor the address response limit
* Limit the number of peers in the address book
* Allow changing the address book limit in tests
* Add tests for the address book length limit
* rustfmt
* Stop checking the entire AddressBook for each connection attempt
* Stop redundant peer time checks within the address book
* Stop calling `Instant::now` 3 times for each address book update
* Only get the time once each time an address book method is called
* Update outdated comment
* Use an OrderedMap to efficiently store address book peers
* Add address book order tests
* Tweak a log message
* Only retry failed DNS once, then use the other DNS responses
* Limit broadcasts to half the peers
* Use a longer minimum interval for GetAddr requests
* Reduce the syncer and mempool crawler fanouts
* Stop resetting the mempool twice when it starts up
This spawns two crawlers, which send two fanouts,
so it can use up a lot of peers.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Download and load Sprout parameters using zcash_proofs
Also update some librustzcash dependencies, to avoid duplicate dependencies.
* Update upstream orchard to avoid a compilation error
* Skip librustzcash batch refactor for now, to avoid compilation errors
* Change the cache ID, so we actually cache Sprout
* Move existing file checks into zcash_proofs
* Add a 1 hour timeout to parameter file downloads
* Give other tasks priority, before spawning the download task
* Update to the latest version of our modified librustzcash fork
* Change the cache key for Sprout
* Add 40 minutes to CI timeouts for occasional sprout downloads
* Update to zcash_proofs with split downloads
* Check file sizes to help debug parameter load failures in zcash_proofs
* Start the second download once the first has finished in zcash_proofs
* Document the parameter download task
* Stop hashing existing files twice
* Start network before verifiers
This makes the Groth16 download task start as late as possible.
* Explain why the Groth16 download must happen first
* Speed up Zebra shutdown: skip waiting for the tokio runtime
* Replace Zcash parameters crates with pre-downloaded local parameter files
* Download Zcash parameters using the `zcashd` script in CI and Docker
* Add a zcash_proofs dependency to zebra-consensus
* Download Sapling parameters using zcash_proofs, rather than fetch-params.sh
* Add a new `zebrad download` subcommand
This command isn't required for nomrmal usage.
But it's useful when testing, or launching multiple Zebra instances.
* Use `zebrad download` in CI to pre-download parameters
* Log a helpful hint if downloading fails
* Allow some duplicate dependencies currently hidden by orchard
* Spawn a separate task to download Groth16 parameters
* Run the parameter download with code coverage
This avoids re-compining Zebra with and without coverage.
* Update Cargo.lock after rebase
* Try to pass `download` as an argument to `zebrad` in coverage CI
* Fix copy and paste comment typos
* Add path and download examples, like zcash_proofs
* Download params in CI just like zcash_proofs does
* Delete a redundant build step
* Implement graceful shutdown for zebrad start
* Send coverage summary to /dev/null when getting the params path
* Use the correct parameters path and download commands in CI
* Explain pre-downloads
* Avoid calling params_folder twice
* Rename parameter types and methods for consistency
```sh
fastmod SaplingParams SaplingParameters zebra*
fastmod Groth16Params Groth16Parameters zebra*
fastmod PARAMS GROTH16_PARAMETERS zebra*
fastmod params_folder directory zebra*
```
And a manual variable name tweak.
* rustfmt
* Remove a redundant coverage step
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Implement graceful shutdown for the peer set
* Use the minimum lookahead limit in acceptance tests
* Enable a doctest that compiles with newly public modules
* Revert "Remove commented-out code"
This reverts commit 9e69777925f103ee11e5940bba95b896c828839b.
* Implement deserialization for `addrv2` messages
* Limit addr and addrv2 messages to MAX_ADDRS_IN_MESSAGE
* Clarify address version comments
* Minor cleanups and fixes
* Add preallocation tests for AddrV2
* Add serialization tests for AddrV2
* Use prop_assert in AddrV2 proptests
* Use a generic utility method for deserializing IP addresses in `addrv2`
* Document the purpose of a conversion to MetaAddr
* Fix a comment typo, and clarify that comment
* Clarify the unsupported AddrV2 network ID error and enum variant names
```sh
fastmod AddrV2UnimplementedError UnsupportedAddrV2NetworkIdError zebra-network
fastmod Unimplemented Unsupported zebra-network
```
* Fix and clarify unsupported AddrV2 comments
* Replace `panic!` with `unreachable!`
* Clarify a comment about skipping a length check in a test
* Remove a redundant test
* Basic addr (v1) and addrv2 deserialization tests
* Test deserialized IPv4 and IPv6 values in addr messages
* Remove redundant io::Cursor
* Add comments with expected values of address test vectors
* Update `tower` to version `0.4.9`
Update to latest version to add support for Tokio version 1.
* Replace usage of `ServiceExt::ready_and`
It was deprecated in favor of `ServiceExt::ready`.
* Update Tokio dependency to version `1.13.0`
This will break the build because the code isn't ready for the update,
but future commits will fix the issues.
* Replace import of `tokio::stream::StreamExt`
Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.
* Use `IntervalStream` in `zebra-network`
In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.
* Use `IntervalStream` in `inventory_registry`
In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.
* Use `BroadcastStream` in `inventory_registry`
In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.
* Handle `Semaphore::acquire` error in `tower-batch`
Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.
* Handle `Semaphore::acquire` error in `zebrad` test
On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.
* Update some `zebra-network` dependencies
Use versions compatible with Tokio version 1.
* Upgrade Hyper to version 0.14
Use a version that supports Tokio version 1.
* Update `metrics` dependency to version 0.17
And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.
* Use `f64` as the histogram data type
`u64` isn't supported as the histogram data type in newer versions of
`metrics`.
* Update the initialization of the metrics component
Make it compatible with the new version of `metrics`.
* Simplify build version counter
Remove all constants and use the new `metrics::incement_counter!` macro.
* Change metrics output line to match on
The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.
* Update `sentry` to version 0.23.0
Use a version compatible with Tokio version 1.
* Remove usage of `TracingIntegration`
This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.
* Add sentry layer to tracing initialization
This seems like the replacement for `TracingIntegration`.
* Remove unnecessary conversion
Suggested by a Clippy lint.
* Update Cargo lock file
Apply all of the updates to dependencies.
* Ban duplicate tokio dependencies
Also ban git sources for tokio dependencies.
* Stop allowing sentry-tracing git repository in `deny.toml`
* Allow remaining duplicates after the tokio upgrade
* Use C: drive for CI build output on Windows
GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.
Co-authored-by: teor <teor@riseup.net>
* Fix some links and linewraps
* Flesh out module-level rustdoc for mempool component
* Links for Downloads and Storage
* Tidy rustdoc links
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Avoid spurious test failures on busy machines
* Revert "Avoid spurious test failures on busy machines"
This reverts commit 4430a48ef3.
* Increase request delay for mock network service
This is necessary because some tests were having timeouts when running
on macOS for the CI. This meant that it took longer than expected for
the subject under test to end up sending the network request.
Co-authored-by: teor <teor@riseup.net>
* Tidy links and notes for some mempool rustdoc
* Fix link
* Fix some links and linewraps
* minor spacing
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Limit the size and age of the ZIP-401 rejected transaction ID list
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix bug in EvictionList; improve documentation
* Separate public and non-public parts of the documentation
* Fix tests
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix bug in EvictionList::len()
* Make EvictionList::len() mutable; prune the list inside it
* Limit the size of EvictedList::ordered_entries
* Increase eviction_list_time_mixed time constants to try to make it pass on MacOS
* Simplify logic by assuming refreshes will never happen
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Compiling fixes
* Remove MEMPOOL_SIZE and just rely on the ZIP-401 cost limit
Co-authored-by: teor <teor@riseup.net>
* ZIP-401 weighted random mempool eviction
* rename zcash.mempool.total_cost.bytes to zcash.mempool.cost.bytes
Co-authored-by: teor <teor@riseup.net>
* Remove duplicated lines
* Add cost() method to UnminedTx
Update serialization failure messages
* More docs quoting ZIP-401 rules
* Change mempool::Storage::new() to handle Copy-less HashMap, HashSet
* mempool: tidy cost types and evict_one()
* More consensus rule docs
* Refactor calculating mempool costs for Unmined transactions
* Add a note on asympotic performance of calculating weights of txs in mempool
* Bump test mempool / storage config to avoid weighted random cost limits
* Use mempool tx_cost_limit = u64::MAX for some tests
* Remove failing tests for now
* Allow(clippy::field-reassign-with-default) because of a move on a type that doesn't impl Copy
* Fix mistaken doctest formatting
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Increase test timeout for Windows builds
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Try simulating a chain growth
* Adjust the transaction expiry height
The mempool evicts expired transactions. When working with mocked data,
appending a new block typically clears the mempool because transactions become
expired. For this reason, the expiry height of each transactions is adjusted so
that it is greater than the new chain tip's height.
* Refactor the code so that it works with `VerifiedUnminedTx`
* Fix a typo
* Fix clippy warnings
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
There are a lot of these messages when Zebra starts up.
They might be slowing down CI and causing timeouts.
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Create a `NowOrLater` helper type
A replacement for `FutureExt::now_or_never` that ensures that the task
is scheduled for waking up later when the inner future is ready.
* Use `NowOrLater` to fix possible delay bug
Previous usage of `now_or_never` meant that the underlying task wasn't
being scheduled to awake when the `Downloads` stream produced a new
item. Using `NowOrLater` instead fixes that issue.
* Ignore AlreadyInChain error in the syncer
* Split Cancelled errors; add them to should_restart_sync exceptions
* Also filter 'block is already comitted'; try to detect a wrong downcast
* Rename tx downloader & verifier metrics
* Add version to mempool metrics
* Add new metrics
* Make sure mempool gauges are zeroed when instances are dropped
* Updated mempool grafana dashboard
* Removed transaction verification dashboard; moved to mempool
* Update mempool dashboard
* Add reason to error labels in mempool dashboard
* Rename some metrics per review
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Guarantee unique IDs in mempool service responses
* Guarantee unique IDs in crawler task mempool Queue requests
Also update the tests to use unique IDs.
* Add a CheckForVerifiedTransactions mempool request
Also document the mempool request and response variants.
* Spawn a QueueChecker task to check for newly verified transactions
This task makes sure that transactions reliably propagate,
rather than relying on peer requests or responses to trigger propagation.
* Update the start command documentation
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Impl Drop, Default and take() for ActiveState
* Refactor Mempool::poll_ready to check disabled and reset first
Also remove some levels of nesting.
* Use the same code for dropping and resetting the mempool
* Document where the tasks are dropped when switching states
* Log mempool resets at info level
And add heights to mempool enable/disable/reset logs
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Guarantee unique IDs in mempool service responses
* Guarantee unique IDs in crawler task mempool Queue requests
Also update the tests to use unique IDs.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Arya
teor
Alfredo Garcia
dependabot[bot]
Conrado Gouvea
Marek
Janito Vaqueiro Ferreira Filho
Dimitris Apostolou
Deirdre Connolly
Deirdre Connolly