* Patch equihash to use the solver branch
* Add an internal-miner feature and set up its dependencies
* Remove 'Experimental' from mining RPC docs
* Fix a nightly clippy::question_mark lint
* Move a byte array utility function to zebra-chain
* fixup! Add an internal-miner feature and set up its dependencies
* Add an equihash::Solution::solve() method with difficulty checks
* Check solution is valid before returning it
* Add a TODO to check for peers before mining
* Move config validation into GetBlockTemplateRpcImpl::new()
* fixup! fixup! Add an internal-miner feature and set up its dependencies
* Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls
* Start adding an internal miner component
* Add the miner task to the start command
* Add basic miner code
* Split out a method to mine one block
* Spawn to a blocking thread
* Wait until a valid template is available
* Handle shutdown
* Run mining on low priority threads
* Ignore some invalid solutions
* Use a difference nonce for each solver thread
* Update TODOs
* Change the patch into a renamed dependency to simplify crate releases
* Clean up instrumentation and TODOs
* Make RPC instances cloneable and clean up generics
* Make LongPollId Copy so it's easier to use
* Add API to restart mining if there's a new block template
* Actually restart mining if there's a new block template
* Tidy instrumentation
* fixup! Move config validation into GetBlockTemplateRpcImpl::new()
* fixup! Make RPC instances cloneable and clean up generics
* Run the template generator and one miner concurrently
* Reduce logging
* Fix a bug in getblocktemplate RPC tip change detection
* Work around some watch channel change bugs
* Rate-limit template changes in the receiver
* Run one mining solver per available core
* Use updated C code with double-free protection
* Update to the latest solver branch
* Return and submit all valid solutions
* Document what INPUT_LENGTH means
* Fix watch channel change detection
* Don't return early when a mining task fails
* Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness
* Make existing parallelism docs and configs consistent
* Add a mining parallelism config
* Use the minimum of the configured or available threads for mining
* Ignore optional feature fields in tests
* Downgrade some frequent logs to debug
* Document new zebrad features and tasks
* Describe the internal-miner feature in the CHANGELOG
* Update dependency to de-duplicate equihash solutions
* Use futures::StreamExt instead of TryStreamExt
* Fix a panic message typo
* Fix availability of tokio::time in scanner
* Create a function that parses a key into a list of keys
* Pass a ChainTipChange to the scanner function
* Convert a scanned block to a sapling result
* Make it easier to pass keys and blocks
* Increase scanner wait times
* Parse keys once at the start of the scan
* Get a block from the state instead of the tip
* Don't log secret keys, only log every 100,000 blocks
* Scan each block and add the results to storage
* Move blocking tasks into spawn_blocking()
* Update the acceptance test
* Use a dummy sapling tree size
* Use a larger dummy size
* Expose IntoDisk and FromDisk in zebra-state
* Implement database serialization for SaplingScanningKey Strings
* Implement serialization for Vec<SaplingScannedResult> (Vec<transaction::Hash>)
* Implement seralization for SaplingScannedDatabaseIndex
* Add an is_empty() method
* Add a read method for a specific index, and document it
* Implement writing scanner results to the database
* Make read name more explicit
* Implement writing scanner keys
* Implement reading sapling keys
* Spawn blocking tasks correctly in async code
* Change storage results methods to use the database
* Update tests that use storage
* Use spawn_blocking() for database methods
* Change the check interval to slightly less than the block interval
* Expose raw database methods with shielded-scan
* fix `scan_task_starts` test
* minor doc change in test
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Create an empty storage/db module
* Use ephemeral storage in tests
* Populate storage inside new() method
* Move scanner setup into an init() method
* Pass the network to scanner init
* Create a database but don't actually use it
* Skip shutdown format checks when skipping format upgrades
* Allow the scanner to skip launching format upgrades in production
* Refactor skipping format upgrades so it is consistent
* Allow checking configs for equality
* Restore Network import
* Provide access to DiskDb and DiskWriteBatch outside the state using a feature
* Actually let's export ZebraDb for the format upgrade code
* Pass column families to ZebraDb as an argument
* Allow the database kind to be changed in config.rs
* Use the state kind in finalized_state.rs
* Allow different database kinds in ZebraDb, but don't move the upgrade code yet
* Allow different database kinds in DiskDb
* Allow different database kinds in upgrade.rs, but don't split the upgrade code out yet
* Add new arguments to raw database tests
* Fix doc links
* Fix internal imports
* Fix unused code
* Update zebrad version metadata
* Create a specific state database delete function
* Fix state exports
* Fix zebrad tests
* Fix zebrad state write tests
* Make CI run again
* Fix dead code warnings for test methods
* Remove unnecessary async on some tests
* Fix logging required by tests
* Fix logging required in test itself
* Fix variable names
* Try to copy the message and add regexes
* connect zebrad with zebra-scan
* remove unwrap
* use tokio::sleep
* fix the task handler
* Don't panic on an empty state
---------
Co-authored-by: teor <teor@riseup.net>
* De-duplicate app_version and user_agent code, rename to build_version
* Make RPC testnet flag forward-compatible with additional testnets
* Fix RPC tests with new argument
* Use "modified" rather than "dirty" for uncommitted changes in build metadata
* Split the vergen version into its own function
* Increase concurrency limit, reduce peer broadcast
* Fix a div_ceil() TODO
* Document security requirements of inbound peer overload handling
* Reduce drop probability and fix its formatting
* Put a 5 second timeout on inbound service requests
* Update drop probability tests
* Add error types, metrics, and logging for InboundTimeout errors
* updates Cargo.toml
* Migrate to abscissa 0.7.0
* Avoid panic from calling color_eyre::install twice
* Uses 'start' as the default subcommand
* updates default cmd logic
* Fixes minor cli issues
* removes outdated check in acceptance test
* Adds a test for process_cli_args, fixes version_args test.
Adds -V to process_cli_args match case
* Revert "fix(clippy): Silence future-incompat warnings until we upgrade Abscissa (#6024)"
This reverts commit dd90f79b48.
* Drops the worker guard to flush logs when zebra shuts down
* Adds cargo feature to clap
* restores process_cli_args
* updates deny.toml
* Updates EntryPoint help template
* Updates subcommand help msgs
* removes trailing whitespace, capitalizes sentences
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* revert parts of revert "Revert fix(clippy): Silence future-incompat warnings until we upgrade Abscissa"
* Applies suggestions from code review
* Moves EntryPoint to its own module
* fixes version_args test
* Updates changelog
* Prunes redundant test cases
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Revert "Prunes redundant test cases"
This reverts commit 3f73979184.
* Update zebrad/src/commands/entry_point.rs
Co-authored-by: teor <teor@riseup.net>
* Add missing import
* Updates `process_cli_args` to return a result
---------
Co-authored-by: teor <teor@riseup.net>
* Rewrite some state cache docs to clarify
* Add a zebra_network::Config.cache_dir for peer address caches
* Add new config test files and fix config test failure message
* Create some zebra-chain and zebra-network convenience functions
* Add methods for reading and writing the peer address cache
* Add cached disk peers to the initial peers list
* Add metrics and logging for loading and storing the peer cache
* Replace log of useless redacted peer IP addresses
* Limit the peer cache minimum and maximum size, don't write empty caches
* Add a cacheable_peers() method to the address book
* Add a peer disk cache updater task to the peer set tasks
* Document that the peer cache is shared by multiple instances unless configured otherwise
* Disable peer cache read/write in disconnected tests
* Make initial peer cache updater sleep shorter for tests
* Add unit tests for reading and writing the peer cache
* Update the task list in the start command docs
* Modify the existing persistent acceptance test to check for peer caches
* Update the peer cache directory when writing test configs
* Add a CacheDir type so the default config can be enabled, but tests can disable it
* Update tests to use the CacheDir config type
* Rename some CacheDir internals
* Add config file test cases for each kind of CacheDir config
* Panic if the config contains invalid socket addresses, rather than continuing
* Add a network directory to state cache directory contents tests
* Add new network.cache_dir config to the config parsing tests
* rename verifiers
* rename `PreparedBlock` to `SemanticallyVerifiedBlock`
* rename `CommitBlock` to `SemanticallyVerifiedBlock`
* rename `FinalizedBlock` to `CheckpointVerifiedBlock`
* rename `CommitFinalizedBlock` to `CommitCheckpointVerifiedBlock`
* rename `FinalizedWithTrees` to `ContextuallyVerifiedBlockWithTrees`
* rename `ContextuallyValidBlock` to `ContextuallyVerifiedBlock`
* change some `finalized` variables or function arguments to `checkpoint_verified`
* fix docs
* document the difference between `CheckpointVerifiedBlock` and `ContextuallyVerifiedBlock`
* fix doc links
* apply suggestions to request
Co-authored-by: Marek <mail@marek.onl>
* apply suggestions to service
Co-authored-by: Marek <mail@marek.onl>
* apply suggestions to finalized_state.rs and write.rs
Co-authored-by: Marek <mail@marek.onl>
* fmt
* change some more variable names
* change a few missing generics
* fix checkpoint log issue
* rename more `prepared` vars `semantically_verified`
* fix test regex
* fix test regex 2
---------
Co-authored-by: Marek <mail@marek.onl>
* add user agent as argument, use git to auto build zebra user agent
* try to fix test
* fix typo
* change expect text
* remove newline
* fix some docs
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* refuse to run Zebra if it is too old
* update the release checklist to consider the constants
* bring newline back
* apply new end of support code
* attempt to add tests (not working yet)
* move eos to progress task
* move tests
* add acceptance test (not working)
* fix tests
* change to block height checks (ugly code)
* change warn days
* refactor estimated blocks per day, etc
* move end of support code to its own task
* change test
* fix some docs
* move constants
* remove uneeded conversions
* downgrade tracing
* reduce end of support time, fix ci changing debugs to info again
* update instructions
* add failure messages
* cargo lock update
* unify releaase name constant
* change info msg
* clippy fixes
* add a block explorer
* ignore testnet in end of support task
* change panic to 16 weeks
* add some documentation about end of support
* Tweak docs wording
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Silence an extremely verbose error in zebra-consensus tests
This disables around 10,000 logs like:
2023-04-18T02:46:28.441662Z WARN init{config=Config { checkpoint_sync: true, debug_skip_parameter_preload: false } network=Mainnet debug_skip_parameter_preload=true}: unexpected error: Closed in state request while verifying previous state checkpoints
* Increase the outbound connection interval to 100ms
* Start the inbound service as soon as possible, and the syncer last
* Increase acceptance test time limits to get more debug info
* Add more debug info to inbound service overload tracing messages
* Implement Display and to_string() for NetworkUpgrade
* Add a progress-bar feature to zebrad
* Add the progress bar writer to the tracing component
* Add a block progress bar transmitter
* Correctly shut down the progress bar, and shut it down on an interrupt
* Make it clearer that the progress task never exits
* Add a config for writing logs to a file
* Add a progress-bar feature to zebra-network
* Add a progress bar for the address book size
* Add progress bars for never attempted and failed peers
* Add an optional limit and label to connection counters
* Add open connection progress bars
* Improve CheckpointList API and CheckpointVerifier debugging
* Add checkpoint index and checkpoint queue progress bars
* Security: Limit the number of non-finalized chains tracked by Zebra
* Make some NonFinalizedState methods available with proptest-impl
* Add a non-finalized chain count progress bar
* Track the last fork height for newly forked chains
* Add a should_count_metrics to Chain
* Add a display method for PartialCumulativeWork
* Add a progress bar for each chain fork
* Add a NonFinalizedState::disable_metrics() method and switch to using it
* Move metrics out of Chain because we can't update Arc<Chain>
* Fix: consistently use best chain order when searching chains
* Track Chain progress bars in NonFinalizedState
* Display work as bits, not a multiple of the target difficulty
* Handle negative fork lengths by reporting "No fork"
* Correctly disable unused fork bars
* clippy: rewrite using `match _.cmp(_) { ... }`
* Initial mempool progress bar implementation
* Update Cargo.lock
* Add the actual transaction size as a description to the cost bar
* Only show mempool progress bars after first activation
* Add queued and rejected mempool progress bars
* Clarify cost note is actual size
* Add tracing.log_file config and progress-bar feature to zebrad docs
* Derive Clone for Chain
* Upgrade to howudoin 0.1.2 and remove some bug workarounds
* Directly call the debug formatter to Display a Network
Co-authored-by: Arya <aryasolhi@gmail.com>
* Rename the address count metric to num_addresses
Co-authored-by: Arya <aryasolhi@gmail.com>
* Simplify reverse checkpoint lookup
Co-authored-by: Arya <aryasolhi@gmail.com>
* Simplify progress bar shutdown code
Co-authored-by: Arya <aryasolhi@gmail.com>
* Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE
* Document that the progress task runs forever
* Fix progress log formatting
* If progress-bar is on, log to a file by default
* Create missing directories for log files
* Add file security docs for running Zebra with elevated permissions
* Document automatic log file, spell progress-bar correctly
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Allow missing docs directly on derived error types
* Make Request::BestChainBlockHash redirect to the ReadStateService
* Re-write the checkpoint_sync documentation based on the latest consensus rules
* Expose the underlying iterator for CheckpointList
* Validate existing state block hashes at startup, but ignore the result
* Monitor state block hash checkpoint task in the start command
* Fix indentation
* Make logging consistent
* Explain the config needed for full security
* Tidy required checkpoints docs, expand other docs
* Add security and deprecation changelog entries
* Replace task handle vector with a struct
* Add a test that this consensus-critical code actually runs and finishes
* Make some state methods and types available in tests
* Add missing docs
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* adds ValidateBlock request to state
* adds `Request` enum in block verifier
skips solution check for BlockProposal requests
calls CheckBlockValidity instead of Commit block for BlockProposal requests
* uses new Request in references to chain verifier
* adds getblocktemplate proposal mode response type
* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state
* Adds PR review revisions
* adds info log in CheckBlockProposalValidity
* Reverts replacement of match statement
* adds `GetBlockTemplate::capabilities` fn
* conditions calling checkpoint verifier on !request.is_proposal
* updates references to validate_and_commit_non_finalized
* adds snapshot test, updates test vectors
* adds `should_count_metrics` to NonFinalizedState
* Returns an error from chain verifier for block proposal requests below checkpoint height
adds feature flags
* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD
* adds back block::Request to zebra-consensus lib
* updates snapshots
* Removes unnecessary network arg
* skips req in tracing intstrument for read state
* Moves out block proposal validation to its own fn
* corrects `difficulty_threshold_is_valid` docs
adds/fixes some comments, adds TODOs
general cleanup from a self-review.
* Update zebra-state/src/service.rs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: teor <teor@riseup.net>
* check best chain tip
* Update zebra-state/src/service.rs
Co-authored-by: teor <teor@riseup.net>
* Applies cleanup suggestions from code review
* updates gbt acceptance test to make a block proposal
* fixes json parsing mistake
* adds retries
* returns reject reason if there are no retries left
* moves result deserialization to RPCRequestClient method, adds docs, moves jsonrpc_core to dev-dependencies
* moves sleep(EXPECTED_TX_TIME) out of loop
* updates/adds info logs in retry loop
* Revert "moves sleep(EXPECTED_TX_TIME) out of loop"
This reverts commit f7f0926f4050519687a79afc16656c3f345c004b.
* adds `allow(dead_code)`
* tests with curtime, mintime, & maxtime
* Fixes doc comment
* Logs error responses from chain_verifier CheckProposal requests
* Removes retry loop, adds num_txs log
* removes verbose info log
* sorts mempool_txs before generating merkle root
* Make imports conditional on a feature
* Disable new CI tests until bugs are fixed
* adds support for getpeerinfo RPC
* adds vector test
* Always passes address_book to RpcServer
* Update zebra-network/src/address_book.rs
Co-authored-by: teor <teor@riseup.net>
* Renames PeerObserver to AddressBookPeers
* adds getpeerinfo acceptance test
* Asserts that addresses parse into SocketAddr
* adds launches_lightwalletd field to TestType::LaunchWithEmptyState and uses it from the get_peer_info test
* renames peer_observer mod
* uses SocketAddr as `addr` field type in PeerInfo
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Log zebrad config path as well as config values
* Remove duplicate config log
* Only log config path and data for server commands
* Use config ref instead
* Make the queue runner task shut down when the RpcImpl is dropped
* Move RPC server startup into the spawn() tokio future
* Return a shutdown handle from the RPC spawn() method
* Shut down the RPC server properly when Zebra shuts down
* Add a changelog entry for this security fix
* Call RpcServer::shutdown() when it is dropped, and wait
* Block on RPC server shutdown when Zebra's tasks have an error
* Add a getblocktemplate-rpcs feature to zebra-chain, and fix missing feature deps
* Add a coinbase transaction creation stub
* Add coinbase creation to zebra-chain
* Add coinbase creation and miner subsidy to zebra-consensus
* Add the miner config to the GetBlockTemplateRpcImpl
* Generate the coinbase transaction in the getblocktemplate RPC
* Provide fake valid block heights to getblocktemplate RPC tests
* Update getblocktemplate RPC snapshots
* Add a getblocktemplate.coinbase_tx deserialized transaction snapshot test
* Update snapshots
* Return funding stream outputs in the same order every time
* Update snapshots
* Fix a script bytes bug
* Update snapshots
* adds submitblock rpc method
* re-orders imports
* replaces thread::yield_now with async yield_now
* Fix doc warnings and unused variable warnings, add missing docs
* Mark work_id as optional
* Use the same ChainVerifier for downloaded and submitted blocks
* Revert unused changes & minor cleanups
* Document currently-unreachable code
* updates tests and submit_block response for AlreadyVerified error
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* changes names from BlockVerifier to ChainVerifier and block_verifier to chain_verifier to keep it consistent with naming in zebra-consensus
* move how to run the submit_block test example to acceptance.rs
* updates snapshot tests
* moved acceptance test to a separate file
* removes extra tower::ServiceBuilder::new(), updates docs
* updates vectors and snapshot tests, changes hex decoding error in submit_block method from server error to parse error
* hides errors module in zebra-rpc behind a feature flag and adds docs.
* Updates snapshot test, adds mod docs, moves HexData to its own mod, and removes the unrelated make_server_error refactor for now
* update submit block acceptance test mod doc
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a new block commit task and channels, that don't do anything yet
* Add last_block_hash_sent to the state service, to avoid database accesses
* Update last_block_hash_sent regardless of commit errors
* Rename a field to StateService.max_queued_finalized_height
* Commit finalized blocks to the state in a separate task
* Check for panics in the block write task
* Wait for the block commit task in tests, and check for errors
* Always run a proptest that sleeps once
* Add extra debugging to state shutdowns
* Work around a RocksDB shutdown bug
* Close the finalized block channel when we're finished with it
* Only reset state queue once per error
* Update some TODOs
* Add a module doc comment
* Drop channels and check for closed channels in the block commit task
* Close state channels and tasks on drop
* Remove some duplicate fields across StateService and ReadStateService
* Try tweaking the shutdown steps
* Update and clarify some comments
* Clarify another comment
* Don't try to cancel RocksDB background work on drop
* Fix up some comments
* Remove some duplicate code
* Remove redundant workarounds for shutdown issues
* Remode a redundant channel close in the block commit task
* Remove a mistaken `!force` shutdown condition
* Remove duplicate force-shutdown code and explain it better
* Improve RPC error logging
* Wait for chain tip updates in the RPC tests
* Wait 2 seconds for chain tip updates before skipping them
* Remove an unnecessary block_in_place()
* Fix some test error messages that were changed by earlier fixes
* Expand some comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
* Actually drop children of failed blocks
* Explain why we drop descendants of failed blocks
* Clarify a comment
* Wait for chain tip updates in a failing test on macOS
* Clean duplicate finalized blocks when the non-finalized state activates
* Send an error when receiving a duplicate finalized block
* Update checkpoint block behaviour, document its consensus rule
* Wait for chain tip changes in inbound_block_height_lookahead_limit test
* Wait for the genesis block to commit in the fake peer set mempool tests
* Disable unreliable mempool verification check in the send transaction test
* Appease rustfmt
* Use clear_finalized_block_queue() everywhere that blocks are dropped
* Document how Finalized and NonFinalized clones are different
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* updates comments, renames send_process_queued, other minor cleanup
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* updates comments, renames send_process_queued, other minor cleanup
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* removes duplicate field definitions on StateService that were a result of a bad merge
* update NotReadyToBeCommitted error message
* Appear rustfmt
* Fix doc links
* Rename a function to initial_contextual_validity()
* Do error tasks on Err, and success tasks on Ok
* Simplify parent_error_map truncation
* Rewrite best_tip() to use tip()
* Rename latest_mem() to latest_non_finalized_state()
```sh
fastmod latest_mem latest_non_finalized_state zebra*
cargo fmt --all
```
* Simplify latest_non_finalized_state() using a new WatchReceiver API
* Expand some error messages
* Send the result after updating the channels, and document why
* wait for chain_tip_update before cancelling download in mempool_cancel_mined
* adds `sent_non_finalized_block_hashes` field to StateService
* adds batched sent_hash insertions and checks sent hashes in queue_and_commit_non_finalized before adding a block to the queue
* check that the `curr_buf` in SentHashes is not empty before pushing it to the `sent_bufs`
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix rustfmt
* Check for finalized block heights using zs_contains()
* adds known_utxos field to SentHashes
* updates comment on SentHashes.add method
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* return early when there's a duplicate hash in QueuedBlocks.queue instead of panicking
* Make finalized UTXOs near the final checkpoint available for full block verification
* Replace a checkpoint height literal with the actual config
* Update mainnet and testnet checkpoints - 7 October 2022
* Fix some state service init arguments
* Allow more lookahead in the downloader, but less lookahead in the syncer
* Add the latest config to the tests, and fix the latest config check
* Increase the number of finalized blocks checked for non-finalized block UTXO spends
* fix(log): reduce verbose logs for block commits (#5348)
* Remove some verbose block write channel logs
* Only warn about tracing endpoint if the address is actually set
* Use CloneError instead of formatting a non-cloneable error
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Increase block verify timeout
* Work around a known block timeout bug by using a shorter timeout
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* use spawn_blocking to run zebra_state::init from start cmd
* uses zebra_state::spawn_init in copy-state command
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* adds start as default subcommand for zebrad
* moves EntryPoint to submodule and adds a test
* moves all start tests to config_test to avoid listener conflicts
* Update zebrad/src/application/entry_point.rs docs
* Revert "moves all start tests to config_test to avoid listener conflicts"
This reverts commit 61ce46f5a1.
* Update based on test API changes from another PR
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Return the maximum checkpoint height from the chain verifier
* Return the verified block height from the sync downloader
* Track the verified height in the syncer
* Use a lower concurrency limit during full verification
* Get the tip from the state before the first verified block
* Limit the number of submitted download and verify blocks in a batch
* Adjust lookahead limits when transitioning to full verification
* Keep unused extra hashes and submit them to the downloader later
* Remove redundant verified_height and state_tip()
* Split the checkpoint and full verify concurrency configs
* Decrease full verification concurrency to 5 blocks
10 concurrent blocks causes 3 minute stalls on some blocks on my machine.
(And it has about 4x as many cores as a standard machine.)
* cargo +stable fmt --all
* Remove a log that's verbose with smaller lookahead limits
* Apply the full verify concurrency limit to the inbound service
* Add a summary of the config changes to the CHANGELOG
* Increase the default full verify concurrency limit to 30
* Improve time logging using humantime
* Only log full seconds, ignore the fractional part
* Move humantime_seconds to tracing::fmt
* Move the progress task to its own module
* Add missing humantime dependency
* Log the network upgrade in progress logs
* Log when Zebra verifies the final checkpoint
* delete old database directories
* check if state directory exists
* skip deleting when ephemeral
* split `check_and_delete_old_databases`
* move `check_and_delete_old_databases` to state
* spawn `check_and_delete_old_databases`
* simplity a bit
* fix(state): only delete old database directories inside the cache directory (#4631)
* Add function comments, tweak log
* Simplify version parsing
* Use spawn_blocking to launch the task on a separate thread, do the cleanup last
* Abort the cleanup task when Zebra exits
* Split directory deletion into its own function, handle ownership
* Rename cache_dir to state_dir
* If an outdated state directory is outside the cache directory, don't delete it
* Minimise diffs
* add test
* fix typos
Co-authored-by: teor <teor@riseup.net>
* add `canonicalize` to test regex
* add another match to test
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Disable the flamegraph feature by default at compile time
* Disable the journald feature by default at compile time
* Also disable inferno dependency, and rearrange features
* Disable the prometheus feature by default at compile time
* Disable the tracing filter reload feature by default at compile time
* Disable tests when corresponding features are disabled
* Add compile-time tracing features to user docs
* Add compile-time features to the metrics user docs
* Document diagnostics as part of the start command tasks and services
* breaking(diagnostics): rename "enable-sentry" feature to "sentry" (#4623)
* Also skip conflict tests when those ports are disabled
* breaking(diagnostics): rename "enable-sentry" feature to "sentry"
This is mostly:
```sh
fastmod enable-sentry sentry
```
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a rpc queue
* Implement the rpc queue
* Add rpc queue tests
* Remove mutex, use broadcast channel
* Have order and limit in the queue
* fix multiple transactions channel
* Use a network argument
* Use chain tip to calculate block spacing
* Add extra time
* Finalize the state check test
* Add a retry test
* Fix description
* fix some docs
* add additional empty check to `Runner::run`
* remove non used method
* ignore some errors
* fix some docs
* add a panic checker to the queue
* add missing file changes for panic checker
* skip checks and retries if height has not changed
* change constants
* reduce the number of queue test cases
* remove suggestion
* change best tip check
* fix(rpc): Check for panics in the transaction queue (#4046)
* Check for panics in the RPC transaction queue
* Add missing pin! and abort in the start task
* Check for transaction queue panics in tests
* Fixup a new RPC test from the main branch
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add the ChainTip and Network to RpcImpl
* Add the new RpcImpl fields to the tests
* Simplify RPC version field using generics
* Temporarily allow unused struct fields
* Use the read-only state service for RPCs
* Refactor non-finalized block lookup into Chain
* Implement the read-only state block request
* Drop the Chain watch channel lock before accessing the finalized state
* Add state service module docs and cleanup
* Move and add finalized state methods
* Add chain and non-finalized state methods
* Cleanup methods and imports
* Create a ReadStateService type
* Add a stub service implementation
* Add a TODO
* Update ReadStateService request stubs with RPC names and tickets
* Documentation updates
* Make RPC State generic bounds accept a buffered state and a read-only state
* Doc updates
* Add missing proptest-impl feature in RPC dev dependencies
* feature(rpc): start adding a `getblock` method
* fix(rpc): replace oneshot
* fix(rpc): replace a panic with error
* fix(rpc): fix test
* feature(rpc): add hex to response
* refactor(rpc): use generic instead of alias
* docs(rpc): improve docs for getblock method
* test(rpc): add a test for getblock method
* deps(rpc): remove non needed tower features
Co-authored-by: teor <teor@riseup.net>
* docs(rpc): add a note to getblock doc
* refactor(rpc): replace alias
* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format
* tests(rpc): add network argument to `populated_state()`
* refactor(rpc): use an error for state service readiness
* fix(rpc): add parameter
* fix(rpc): clippy
* nit(rpc): remove new line from imports
* fix(rpc): remove commented code
* fix(rpc): simplify error
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use a `SerializedBlock` type to help serializing blocks (#3725)
* Create a `SerializedBlock` helper type
Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.
* Use `into_iter` instead of `iter`
There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.
* Use `SerializedBlock` in `GetBlock`
Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.
The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.
* feature(rpc): add suggestions from code review
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* tests(rpc): make sure mempool has no requests in get_block test
* fix(rpc): change height argument type in getblock method
* fix(rpc): rustfmt
* fix(rpc): replace panic
* fix(rpc): change getblock response
* fix(rpc): fix lightwalletd test
* tests(rpc): add a getblock error test
* fix(rpc): try another regex
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Stub `sendrawtransaction` RPC method
Register the RPC method, and stub an implementation that currently just
panics. The method has a single `String` parameter with the hexadecimal
string of the raw transaction's bytes and returns a
`SentTransactionHash` wrapper type that's just a hexadecimal `String` of
the sent transaction's hash.
* Add mempool service instance to `RpcImpl`
Use a type parameter to represent the mempool service using the
interface defined by `zebra-node-services`.
* Update test vector to use a mock mempool service
Update the test to be compatible with the changes to `RpcImpl`. The mock
mempool service is expected to not be used during the test.
* Use a `tower::Buffer` for the mempool service
Make it simpler to send requests to the service in a concurrent manner.
* Return a `Future` from `send_raw_transaction`
Make the call asynchronous.
* Implement `sendrawtransaction` RPC
Deserialize the transaction and send it to be queued for verification
and subsequent inclusion in the mempool.
* Test if mempool receives sent raw transaction
Use a mock service as the mempool service and check that it receives a
sent raw transaction.
* Test using non-hexadecimal string parameter
The method should return an error.
* Test with bytes that fail deserialization
Check that the method returns an invalid parameters error if the input
can't be deserialized as a `Transaction`.
* Test if mempool errors are forwarded to caller
Mempool service errors should be sent back to the remote caller as
server errors.
* Test transactions rejected by the mempool service
Transactions that are rejected by the mempool service should result in
a server error being sent to the caller.
* Improve error message
Add the word "structurally" to make it clear that the issue is in the
transaction's deserialization.
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Add note regarding missing `allowhighfees` param.
The parameter isn't supported yet because `lightwalletd` doesn't use it.
* Update the documentation to be consistent
Follow the convention adopted by the `get_info` RPC method.
* Implement `ToHex` and `FromHex` for `Hash`
Make it easier to generate hexadecimal strings from `transaction::Hash`
instances.
* Use `ToHex` in `Debug` and `Display`
Reduce repeated code.
* Refactor to add `bytes_in_display_order` method
Use it to remove repeated code and improve clarity a bit.
* Use `hex::serialize` to serialize transaction hash
Make the type stricter in its contents, while still serializing the
transaction has as a hexadecimal string.
* Simplify serialization attribute
Deserialization should also use `hex::deserialize`, so using the shorter
attribute makes things easier to read and more future proof.
* Update zebra-chain/src/transaction/hash.rs
* Remove unnecessary lifetime
The anonymous lifetime is automatically inferred by the compiler.
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Stub `sendrawtransaction` RPC method
Register the RPC method, and stub an implementation that currently just
panics. The method has a single `String` parameter with the hexadecimal
string of the raw transaction's bytes and returns a
`SentTransactionHash` wrapper type that's just a hexadecimal `String` of
the sent transaction's hash.
* Add mempool service instance to `RpcImpl`
Use a type parameter to represent the mempool service using the
interface defined by `zebra-node-services`.
* Update test vector to use a mock mempool service
Update the test to be compatible with the changes to `RpcImpl`. The mock
mempool service is expected to not be used during the test.
* Use a `tower::Buffer` for the mempool service
Make it simpler to send requests to the service in a concurrent manner.
* Return a `Future` from `send_raw_transaction`
Make the call asynchronous.
* Implement `sendrawtransaction` RPC
Deserialize the transaction and send it to be queued for verification
and subsequent inclusion in the mempool.
* Test if mempool receives sent raw transaction
Use a mock service as the mempool service and check that it receives a
sent raw transaction.
* Test using non-hexadecimal string parameter
The method should return an error.
* Test with bytes that fail deserialization
Check that the method returns an invalid parameters error if the input
can't be deserialized as a `Transaction`.
* Test if mempool errors are forwarded to caller
Mempool service errors should be sent back to the remote caller as
server errors.
* Test transactions rejected by the mempool service
Transactions that are rejected by the mempool service should result in
a server error being sent to the caller.
* Improve error message
Add the word "structurally" to make it clear that the issue is in the
transaction's deserialization.
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Add note regarding missing `allowhighfees` param.
The parameter isn't supported yet because `lightwalletd` doesn't use it.
* Update the documentation to be consistent
Follow the convention adopted by the `get_info` RPC method.
* Remove mempool service usage line
It contained incomplete information that's not really necessary.
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Fix formatting
`rustfmt` was not executed on the file for the previous commit because I
had edited it on GitHub.
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* tests(rpc): add an rpc endpoint test
* tests(rpc): add an rpc port conflict test
* tests(rpc): refactor some imports
* tests(rpc): fix failures, make test more complete
* tests(rpc): parse json response for better coverage
* tests(rpc): change request
* tests(rpc): wait until port is open in rpc_endpoint test
* tests(rpc): add a delay between launching 2 nodes
* tests(rpc): try 5 seconds
* refactor(rpc): open rpc server faster
* tests(rpc): extend `LAUNCH_DELAY` to 15 seconds
* fix(rpc): disable rpc_conflict test for windows
* fix(ci): skip the RPC tests if the network is disabled
* rustfmt
* fix(zebrad/test): test function return type
* tests(rpc): print server output in assert
* fix(rpc): fix acceptance test looking for string in `build` field
* fix(rpc): reduce the number of acceptable characters in version output
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feature(rpc): add getinfo subversion field and getinfo docs
* feature(rpc): add getinfo build field
* refactor(rpc): replace the lazy_static
* docs(rpc): fic typo, add link to zcashd ticket
* tests(rpc): add getinfo unit test
* docs(rpc): complete comment
* feature(rpc): add an rpc component
* feat(rpc): add a stub for getblockchaininfo
This is the first RPC used by lightwalletd, so we need it for testing.
* fix(rpc): remove non-standard "jsonrpc: 1.0" from lightwalletd
* fix(rpc): re-enable default RPC security checks
* deps(rpc): remove not needed dependency
* fix(rpc): check if RPC task has stopped
* fix(rpc): reduce config by using Option
* fix(rpc): use tokio executor
* security(rpc): turn off rpc by default
* docs(rpc): update a TODO comment
Co-authored-by: teor <teor@riseup.net>
* fix(rpc): blocking tasks
Co-authored-by: teor <teor@riseup.net>
* rename(rpc): rpc.rs to methods.rs
* refactor(rpc): move the server to the zebra-rpc crate
* fix(rpc): clippy derive Default for RPC Config
* fix(dependencies): remove unused dependency features in zebra-rpc
We expect to use all the listed tokio features
to implement and test RPC methods.
* doc(rpc): fix testnet port, add security note
* fix(rpc): change Rust function names and update method doc TODOs
* fix(rpc): add "TODO" to fake RPC responses
* doc(rpc): update module docs
* fix(rpc): simplify server struct derives
* fix(rpc): simplify server code
* doc(rpc): explain how request fixes securely handle user-supplied data
* refactor(rpc): move the compatibility fix to a separate module
* fix(rpc): move the open log inside the spawn, and instrument it
* doc(rpc): fix toml format and provide a config example
Co-authored-by: teor <teor@riseup.net>
* feat(log): log current height when logging sync progress
* fix(test): log the specific error when full sync tests fail
* doc(start): remove an obsolete TODO
We can't decrease this log level, because the tests rely on it.
* fix(test): wait until mempool activates in full sync tests
Changes the log message and log test so that the test only finishes
when the mempool has activated.
There is still a race condition here, between the log timer and
mempool activation. But it should be very rare, because the mempool
is activated immediately when `is_close_to_tip()` becomes true.
* fix(test): warn when Zebra stalls below the maximum checkpoint height
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* feat(log): warn when sync stalls downloading the genesis block
* fix(test): warn when the state hasn't committed a block for a long time
This also improves the full sync tests,
because the warning is checked before logging a successful sync.
* doc(test): update some sync acceptance test comments
* fix(log): use Display formatting to log chrono::Duration
Debug formatting is complicated and hard to read.
* fix(log): stop saying that we've activated the mempool without checking it
We're not checking if the mempool is active, so we can't say that.
* fix(log): minor tidying and TODOs
* fix(doc): fix a typo in the tests
* fix(log): explain the post-checkpoint blocks in progress warning calculations
* fix(doc): explain what could happen if we don't wait for extra blocks
* fix(log): add a percent symbol to a percent log
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Log chain progress while Zebra is syncing
This helps test if the chain tip estimate is accurate,
and helps diagnose problems during full sync tests.
* Update to the latest chain tip estimate API
* fix(state): set state concurrency based on other services' concurrency
* fix(sync): increase the sync downloader lookahead limit
It seems like the recent tokio upgrade made this code even more efficient,
so on testnet we can have around 6000 blocks in flight.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feat(log): log the state tip height as part of sync progress logs
* fix(log): downgrade some verbose state logs to debug
* feat(log): log successful gossiped block verification at info level
These logs help us diagnose slow progress near the tip.
There won't be very many of these logs,
because they only happen near the tip.
* fix(log): spawn top-level tasks within the global Zebra tracing span
* fix(log): spawn blocking top-level tasks within the global Zebra tracing span
Co-authored-by: teor <teor@riseup.net>
teor
Alfredo Garcia
Arya
Janito Vaqueiro Ferreira Filho