* Revert "Update code that we're going to delete in the next PR anyway"
This reverts commit 1fed70da9e.
* Initial zcash-test-block-template script without block proposal construction
* Try creating block data using jq and printf
* Move proposal_block_from_template() to zebra-rpc and remove eyre dependency
* Implement FromStr for DateTime32 and Duration32
* Basic block-template-to-proposal command without time source handling
* Move block proposal code into its own module
* Use time source in block-template-to-proposal
* Make block-template-to-proposal require the getblocktemplate-rpcs feature
* Use block-template-to-proposal in the test script zcash-rpc-block-template-to-proposal
* Apply new hex formatting to commitments and nonces in block proposal tests
* Re-add missing imports from rebase
* Re-add missing conversions from rebase
* Update to new method name after rebase
* Derive a Default impl
* Restore a comment that was accidentally deleted during the rebase
* Avoid a clippy::unwrap-in-result
* Temporarily fix the code for a disabled test
* Fix tool build with Docker caches
* Make clippy happy with the temporary fix
* Give a pass/fail status for each proposal response
* Accept zcashd block templates in block-template-to-proposal
* Fix pretty printing
* Zebra expects a longpollid, so give it a dummy value
* Add "required" fields which Zebra requires
* Use curtime as the dummy maxtime in zcashd templates
* Support large block proposals by reading proposal data from a file
* Test all valid time modes for each proposal
* Allow the user to set the time command
* Put debug logs into their own files
* Exit with an error status when any proposal is invalid
* Log the date and time to make it easier to match errors to node logs
* Derive default using #[default]
* Implement PartialEq manually to satisfy clippy
* Allow a manual derive in test-only code
* Fix some missing docs warnings in the Docker build
* Format commitments and nonces as hex
* Format Sprout and Sapling nullifiers as hex when debugging
* Format Sprout commitments as hex when debugging
* Format redpallas keys as hex when debugging
* Update code that we're going to delete in the next PR anyway
* adds ValidateBlock request to state
* adds `Request` enum in block verifier
skips solution check for BlockProposal requests
calls CheckBlockValidity instead of Commit block for BlockProposal requests
* uses new Request in references to chain verifier
* adds getblocktemplate proposal mode response type
* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state
* Adds PR review revisions
* adds info log in CheckBlockProposalValidity
* Reverts replacement of match statement
* adds `GetBlockTemplate::capabilities` fn
* conditions calling checkpoint verifier on !request.is_proposal
* updates references to validate_and_commit_non_finalized
* adds snapshot test, updates test vectors
* adds `should_count_metrics` to NonFinalizedState
* Returns an error from chain verifier for block proposal requests below checkpoint height
adds feature flags
* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD
* adds back block::Request to zebra-consensus lib
* updates snapshots
* Removes unnecessary network arg
* skips req in tracing intstrument for read state
* Moves out block proposal validation to its own fn
* corrects `difficulty_threshold_is_valid` docs
adds/fixes some comments, adds TODOs
general cleanup from a self-review.
* Update zebra-state/src/service.rs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: teor <teor@riseup.net>
* check best chain tip
* Update zebra-state/src/service.rs
Co-authored-by: teor <teor@riseup.net>
* Applies cleanup suggestions from code review
* updates gbt acceptance test to make a block proposal
* fixes json parsing mistake
* adds retries
* returns reject reason if there are no retries left
* moves result deserialization to RPCRequestClient method, adds docs, moves jsonrpc_core to dev-dependencies
* moves sleep(EXPECTED_TX_TIME) out of loop
* updates/adds info logs in retry loop
* Revert "moves sleep(EXPECTED_TX_TIME) out of loop"
This reverts commit f7f0926f4050519687a79afc16656c3f345c004b.
* adds `allow(dead_code)`
* tests with curtime, mintime, & maxtime
* Fixes doc comment
* Logs error responses from chain_verifier CheckProposal requests
* Removes retry loop, adds num_txs log
* removes verbose info log
* sorts mempool_txs before generating merkle root
* Make imports conditional on a feature
* Disable new CI tests until bugs are fixed
* adds support for getpeerinfo RPC
* adds vector test
* Always passes address_book to RpcServer
* Update zebra-network/src/address_book.rs
Co-authored-by: teor <teor@riseup.net>
* Renames PeerObserver to AddressBookPeers
* adds getpeerinfo acceptance test
* Asserts that addresses parse into SocketAddr
* adds launches_lightwalletd field to TestType::LaunchWithEmptyState and uses it from the get_peer_info test
* renames peer_observer mod
* uses SocketAddr as `addr` field type in PeerInfo
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* adds ValidateBlock request to state
* adds `Request` enum in block verifier
skips solution check for BlockProposal requests
calls CheckBlockValidity instead of Commit block for BlockProposal requests
* uses new Request in references to chain verifier
* adds getblocktemplate proposal mode response type
* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state
* Adds PR review revisions
* adds info log in CheckBlockProposalValidity
* Reverts replacement of match statement
* adds `GetBlockTemplate::capabilities` fn
* conditions calling checkpoint verifier on !request.is_proposal
* updates references to validate_and_commit_non_finalized
* adds snapshot test, updates test vectors
* adds `should_count_metrics` to NonFinalizedState
* Returns an error from chain verifier for block proposal requests below checkpoint height
adds feature flags
* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD
* adds back block::Request to zebra-consensus lib
* updates snapshots
* Removes unnecessary network arg
* skips req in tracing intstrument for read state
* Moves out block proposal validation to its own fn
* corrects `difficulty_threshold_is_valid` docs
adds/fixes some comments, adds TODOs
general cleanup from a self-review.
* Update zebra-state/src/service.rs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: teor <teor@riseup.net>
* check best chain tip
* Update zebra-state/src/service.rs
Co-authored-by: teor <teor@riseup.net>
* Applies cleanup suggestions from code review
* updates gbt acceptance test to make a block proposal
* fixes json parsing mistake
* adds retries
* returns reject reason if there are no retries left
* moves result deserialization to RPCRequestClient method, adds docs, moves jsonrpc_core to dev-dependencies
* moves sleep(EXPECTED_TX_TIME) out of loop
* updates/adds info logs in retry loop
* Revert "moves sleep(EXPECTED_TX_TIME) out of loop"
This reverts commit f7f0926f4050519687a79afc16656c3f345c004b.
* adds `allow(dead_code)`
* tests with curtime, mintime, & maxtime
* Fixes doc comment
* Logs error responses from chain_verifier CheckProposal requests
* Removes retry loop, adds num_txs log
* removes verbose info log
* sorts mempool_txs before generating merkle root
* Make imports conditional on a feature
* Disable new CI tests until bugs are fixed
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* adds ValidateBlock request to state
* adds `Request` enum in block verifier
skips solution check for BlockProposal requests
calls CheckBlockValidity instead of Commit block for BlockProposal requests
* uses new Request in references to chain verifier
* adds getblocktemplate proposal mode response type
* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state
* Adds PR review revisions
* adds info log in CheckBlockProposalValidity
* Reverts replacement of match statement
* adds `GetBlockTemplate::capabilities` fn
* conditions calling checkpoint verifier on !request.is_proposal
* updates references to validate_and_commit_non_finalized
* adds snapshot test, updates test vectors
* adds `should_count_metrics` to NonFinalizedState
* Returns an error from chain verifier for block proposal requests below checkpoint height
adds feature flags
* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD
* adds back block::Request to zebra-consensus lib
* updates snapshots
* Removes unnecessary network arg
* skips req in tracing intstrument for read state
* Moves out block proposal validation to its own fn
* corrects `difficulty_threshold_is_valid` docs
adds/fixes some comments, adds TODOs
general cleanup from a self-review.
* Update zebra-state/src/service.rs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: teor <teor@riseup.net>
* check best chain tip
* Update zebra-state/src/service.rs
Co-authored-by: teor <teor@riseup.net>
* Applies cleanup suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Changelog with trivial entries
* Delete trivial entries
* Summarise known issues in README, but don't change the list yet
* Add block timeouts to known issues
* Update the release template to add missing version files
* Bump crate versions
* Add the required Rust version to the release checklist
* Update the Rust version requirement to 1.65, Zebra now uses `let ... else ...`
* Update checkpoints
* Add checkpoints to the CHANGELOG
* Breaking Rust compiler version change
* Clarify the latest stable supported rust version
* Add some TODOs
* Move and rename height_from_signed_int()
* Move get_block_template() support functions to a submodule
* Fix incorrect P2SH comments and logs
* Split initial checks into their own functions
* Split state fetch into its own function, do some cleanup
* Move get_block_template_opts to get_block_template::parameters
* Fix and simplify test imports
* Rename block_height to next_block_height
* Rename to chain_tip_and_local_time to make it clear what it contains
* Split fetching mempool transactions out, include them in long poll id
* Refactor coinbase generation
* Split default root calculation into a separate function
* Use DateTime32 for getblocktemplate times
* Use typed difficulty fields rather than strings
* Split out a generate coinbase and roots function
* Move GetBlockTemplate construction into a method
* Document what happens to unusual difficulty values
* Clean up some TODOs
* fastmod check_address check_miner_address
* cargo fmt --all
* Fix an incorrect panic message
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move Drop from mempool::ActiveState to mempool::Downloads, to avoid bugs
* Re-verify mempool transactions after a fork
And add a marker struct for mempool download cancellation.
* Update README based on recent mitigations for some issues, tidy format
* Make mempool proptests easier to debug
* Make UnminedTx Display text much smaller
* Update tests for mempool transaction re-verification after forks
* Retry all stored and pending transactions
* Fix a test to check for mempool reset retries
* Declare support for long polling in the getblocktemplate RPC
* Add long polling input and ID structs
* Split out an update_checksum() function
* Implement LongPollId conversion to and from a string
* Use the LongPollId type in the RPC
* Add a longpollid field to the getblocktemplate parameters and responses, but don't use it yet
* Use multiple RPC threads with the getblocktemplate feature, to enable efficient long polling
* Update RPC snapshots
* Remove the "longpoll" capability, it's for miners, not nodes
* Use the long poll length constant in tests
* Update snapshots
* Remove the "long polling is not supported" error
* Fix minor compilation issues after the merge/rebase
* Expand long poll id comments
* Rename estimated height to local height, because that's what it actually is
* Add an invalid params test and fix the long poll id test
* Add modified config for config_tests
* Instrument all the config sub-tests
* Show the missing config file when the test fails
* Fix the generated config file
* Allow a clippy lint in tests
* Explain conversion from bytes to u32
* Remove a duplicate test case
* Remove verbose continuous_blockchain test logs
* Downgrade verbose zebra-network logs to debug
* Downgrade some state logs to debug during tests
* Mark were we would add always-on log filters, if we needed to
* Reduce the number of mempool property tests, to reduce logging
* Box errors to deal with large error warnings, add accessor methods for error properties
* Remove or explain some large enum variant lints
* Turn some tickets into TODOs
* Allow missing docs on single-field error enum variants
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move NoChainTip to a test-only module
* Move test-only ChainSyncStatus code into its own module
* Move RecentSyncLengths mocks to their own module
* Silence an unused test code lint
* Actually, NoChainTip is used in production for isolated connections
* Adds ignored jsonrequestobject argument
* adds docs
returns an error for unsupported options
* returns an error when longpollid is provided
* comments out capabilities enum and uses Vec<String> instead
* fixes spelling mistake
* uncomments GetBlockTemplateCapability enum and uses allow(dead_code) instead
* adds UnknownCapability
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* adds derived traits.
* removes unused [allow(dead_code)]
* test invalid params error for block data
* add capablity to gbt vectors test
* reverts passing in default options to getblocktemplate request in vectors test
* adds a jsonrequestobject with an unknown capability to gbt acceptance test
Co-authored-by: teor <teor@riseup.net>
* Remove a redundant sprout full sync job
* Add two new full sync jobs
* Allow the full sync test to run for 48 hours (estimated current time 40-45 hours)
* Split the conventional fee check into its own method
This will be used for block production and relaying
* Move getblocktemplate transaction selection into a new zip317 module
* Add a block_production_fee_weight field to VerifiedUnminedTx
* Add a custom Zebra minimum transaction weight for block production
* Implement ZIP-317 transaction selection for block production
* Split weighted index setup into its own function
* Split picking a transaction into its own function
* Re-apply: add acceptance test for getblocktemplate method in CI (#5653)
Revert "Revert "change(tests): add acceptance test for getblocktemplate method in CI (#5653)" (#5672)"
This reverts commit 6446e0ec1b.
* Fix incorrect MAX_CONTEXT_BLOCKS assertion in state
* Actually negate the miner fee for the RPC output
* Try the RPC again after waiting for transactions to verify
* Log before the test waits for the mempool to verify transactions
* Use the new ssh key secrets in CI
* updates comments
* adds check nullifier no dup fns for transactions
* Adds:
- check::anchors fn for tx iter
- TODO comments for unifying nullifiers and anchors checks
- new state request
Updates unknown anchor errors to accomodate tx-only check
Calls new state fn from transaction verifier
* updates check::anchors fns to use transactions
updates TransactionContextualValidity request to check sprout anchors
adds comment mentioning TransactionContextualValidity ignores UTXOs
* conditions new state req call on is_mempool
updates tests
* fix doc link / lint error
* checks for duplicate nullifiers with closures
* Update zebra-state/src/service/check/nullifier.rs
Co-authored-by: teor <teor@riseup.net>
* documents find_duplicate_nullifier params
moves if let statement into for loop
* renames new state req/res
* asserts correct response variant in tx verifier
* adds CheckBestChainTipShieldedSpends call in tx verifier to async checks
* re-adds tracing instrumentation to check::anchors fn
renames transaction_in_state to transaction_in_chain
* adds block/tx wrapper fns for anchors checks
* uses UnminedTx instead of transaction.hash()
deletes broken test
* updates new state req/res name
* updates tests and uses par_iter for anchors checks
* Updates check::anchors pub fn docs.
* Adds:
- comments / docs
- a TransactionError variant for ValidateContextError
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* moves downcast to From impl
rustfmt
* moves the ValidateContextError into an Arc
updates comments and naming
* leaves par_iter for another PR
* puts io::Error in an Arc
* updates anchors tests to call tx_anchors check
* updates tests to call tx_no_duplicates_in_chain
slightly improves formatting
* Update zebra-consensus/src/error.rs
Co-authored-by: teor <teor@riseup.net>
* moves Arc from HistoryError to ValidateContextError
Co-authored-by: teor <teor@riseup.net>
* Log zebrad config path as well as config values
* Remove duplicate config log
* Only log config path and data for server commands
* Use config ref instead
* print usage info for --help flag
* update known issues and help command's help text
* Update README.md
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* updates README
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* adds test for getblocktemplate rpc method
* adds the new test to CI
* adds a couple logs
* Adds example for running the test in acceptance.rs
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make the queue runner task shut down when the RpcImpl is dropped
* Move RPC server startup into the spawn() tokio future
* Return a shutdown handle from the RPC spawn() method
* Shut down the RPC server properly when Zebra shuts down
* Add a changelog entry for this security fix
* Call RpcServer::shutdown() when it is dropped, and wait
* Block on RPC server shutdown when Zebra's tasks have an error
* updates mod docs for tests that use future blocks
* updates submitblock test to use TestType methods
* prunes redundant code
* adds check_sync_logs_until
* adds assertion for needs cached state & rpc server
* updates get_raw_future_blocks fn with rpc calls
* updates to get_raw_future_blocks fn and submit_block test
* Rename LightwalletdTestType to TestType
* moves TestType and random_known_rpc_port_config to test_type.rs and config.rs
* moves get_raw_future_blocks to cached_state.rs
* updates ci workflows to include submit block test
* adds get_future_blocks fn and uses it in load_transactions_from_future_blocks
* updates CI docker
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Applies suggestions from code review
* Updates misnamed closure param
* updates mod docs for test_type.rs
Co-authored-by: teor <teor@riseup.net>
* Add a getblocktemplate-rpcs feature to zebra-chain, and fix missing feature deps
* Add a coinbase transaction creation stub
* Add coinbase creation to zebra-chain
* Add coinbase creation and miner subsidy to zebra-consensus
* Add the miner config to the GetBlockTemplateRpcImpl
* Generate the coinbase transaction in the getblocktemplate RPC
* Provide fake valid block heights to getblocktemplate RPC tests
* Update getblocktemplate RPC snapshots
* Add a getblocktemplate.coinbase_tx deserialized transaction snapshot test
* Update snapshots
* Return funding stream outputs in the same order every time
* Update snapshots
* Fix a script bytes bug
* Update snapshots
* Use correct release for getblocktemplate config
* Include at least 2 full checkpoints in the lookahead limit
* Increase full sync timeout to 36 hours
* Only log "synced block height too far ahead of the tip" once
* Replace AboveLookaheadHeightLimit error with pausing the syncer
* Use AboveLookaheadHeightLimit for blocks a very long way from the tip
* Also add the getblocktemplate config, and fix the test message
* Remove an outdated TODO comment
* Allow syncing again when a small number of blocks are in the queue
* Allow some dead code
* Fix error text for state service for syncer
* Fix error handling in syncer
* cargo fmt --all
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* adds submitblock rpc method
* re-orders imports
* replaces thread::yield_now with async yield_now
* Fix doc warnings and unused variable warnings, add missing docs
* Mark work_id as optional
* Use the same ChainVerifier for downloaded and submitted blocks
* Revert unused changes & minor cleanups
* Document currently-unreachable code
* updates tests and submit_block response for AlreadyVerified error
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* changes names from BlockVerifier to ChainVerifier and block_verifier to chain_verifier to keep it consistent with naming in zebra-consensus
* move how to run the submit_block test example to acceptance.rs
* updates snapshot tests
* moved acceptance test to a separate file
* removes extra tower::ServiceBuilder::new(), updates docs
* updates vectors and snapshot tests, changes hex decoding error in submit_block method from server error to parse error
* hides errors module in zebra-rpc behind a feature flag and adds docs.
* Updates snapshot test, adds mod docs, moves HexData to its own mod, and removes the unrelated make_server_error refactor for now
* update submit block acceptance test mod doc
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* allow(clippy::result_large_err)
* Increase the async executor delay expected by tests
* Split getblocktemplate-rpcs OS tests into a separate matrix job
* Add new patch jobs
* allow(unknown_lints)
* Add a legacy_sigop_count field to VerifiedUnminedTx
* Add conversions from Vec<VerifiedUnminedTx> to block header roots
* Add fee and sigops field to block template transactions
* Fix up mempool request names
* Increase existing snapshot test coverage
* Document a new method parameter
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Add documentation for the getblocktemplate RPC
* Add a new mempool::Request::Transactions
* Add conversions from Vec<UnminedTx> to merkle::Root and AuthDataRoot
* Fill in the merkle root and auth data root fields
* Delete the Coinbase type, it's the same as Transaction
* Fill in some other existing types
* Add Hex serialization support to some zebra-chain types
* Add TransactionTemplate fields and fill some in
* Fix test hangs by spawning async tasks
* Add temporary workaround for no transactions in the block
* Encode hashes and roots as hex
* Update RPC snapshots
* Add a missing Request::Transactions handler
* Fix doc warnings
* Fix fee serialization
* Update snapshots for serialization changes
* Add a missing Cargo.lock change
* Change depends type
* Remove duplicate feature entry
* Document the new RPC feature
* Fix a comment typo
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Update default roots docs
* Fix comment typo
* Fix a comment typo
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Arya <aryasolhi@gmail.com>
* adds transactions to ChainTipBlock and rejects transactions from the mempool that have been invalidated by the latest block commit
* merges remove_same_effects in with reject_invalidated_transactions, moves nullifier retrieval to the new Storage method, rejects mined_ids, and updates tests
* updates DuplicateSpend's error message
* fixes tests
* Update zebrad/src/components/mempool/storage.rs
Co-authored-by: teor <teor@riseup.net>
* adds comment
* formatting for the proptest
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* waits for the chain tip update if setup adds mempool transactions and adds match arm for Response::Nil to MempoolTransactionIds request
* updates other match statements checking MempoolTransactionIds requests
* removes wait for chain tip update and waits for AdvertiseBlock earlier instead
* Increment Zebra versions
* Initial draft changelog
* Add blog post to the release checklist
* Say "user testing"
Co-authored-by: teor <teor@riseup.net>
* Add a new block commit task and channels, that don't do anything yet
* Add last_block_hash_sent to the state service, to avoid database accesses
* Update last_block_hash_sent regardless of commit errors
* Rename a field to StateService.max_queued_finalized_height
* Commit finalized blocks to the state in a separate task
* Check for panics in the block write task
* Wait for the block commit task in tests, and check for errors
* Always run a proptest that sleeps once
* Add extra debugging to state shutdowns
* Work around a RocksDB shutdown bug
* Close the finalized block channel when we're finished with it
* Only reset state queue once per error
* Update some TODOs
* Add a module doc comment
* Drop channels and check for closed channels in the block commit task
* Close state channels and tasks on drop
* Remove some duplicate fields across StateService and ReadStateService
* Try tweaking the shutdown steps
* Update and clarify some comments
* Clarify another comment
* Don't try to cancel RocksDB background work on drop
* Fix up some comments
* Remove some duplicate code
* Remove redundant workarounds for shutdown issues
* Remode a redundant channel close in the block commit task
* Remove a mistaken `!force` shutdown condition
* Remove duplicate force-shutdown code and explain it better
* Improve RPC error logging
* Wait for chain tip updates in the RPC tests
* Wait 2 seconds for chain tip updates before skipping them
* Remove an unnecessary block_in_place()
* Fix some test error messages that were changed by earlier fixes
* Expand some comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
* Actually drop children of failed blocks
* Explain why we drop descendants of failed blocks
* Clarify a comment
* Wait for chain tip updates in a failing test on macOS
* Clean duplicate finalized blocks when the non-finalized state activates
* Send an error when receiving a duplicate finalized block
* Update checkpoint block behaviour, document its consensus rule
* Wait for chain tip changes in inbound_block_height_lookahead_limit test
* Wait for the genesis block to commit in the fake peer set mempool tests
* Disable unreliable mempool verification check in the send transaction test
* Appease rustfmt
* Use clear_finalized_block_queue() everywhere that blocks are dropped
* Document how Finalized and NonFinalized clones are different
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* updates comments, renames send_process_queued, other minor cleanup
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* updates comments, renames send_process_queued, other minor cleanup
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* removes duplicate field definitions on StateService that were a result of a bad merge
* update NotReadyToBeCommitted error message
* Appear rustfmt
* Fix doc links
* Rename a function to initial_contextual_validity()
* Do error tasks on Err, and success tasks on Ok
* Simplify parent_error_map truncation
* Rewrite best_tip() to use tip()
* Rename latest_mem() to latest_non_finalized_state()
```sh
fastmod latest_mem latest_non_finalized_state zebra*
cargo fmt --all
```
* Simplify latest_non_finalized_state() using a new WatchReceiver API
* Expand some error messages
* Send the result after updating the channels, and document why
* wait for chain_tip_update before cancelling download in mempool_cancel_mined
* adds `sent_non_finalized_block_hashes` field to StateService
* adds batched sent_hash insertions and checks sent hashes in queue_and_commit_non_finalized before adding a block to the queue
* check that the `curr_buf` in SentHashes is not empty before pushing it to the `sent_bufs`
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix rustfmt
* Check for finalized block heights using zs_contains()
* adds known_utxos field to SentHashes
* updates comment on SentHashes.add method
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* return early when there's a duplicate hash in QueuedBlocks.queue instead of panicking
* Make finalized UTXOs near the final checkpoint available for full block verification
* Replace a checkpoint height literal with the actual config
* Update mainnet and testnet checkpoints - 7 October 2022
* Fix some state service init arguments
* Allow more lookahead in the downloader, but less lookahead in the syncer
* Add the latest config to the tests, and fix the latest config check
* Increase the number of finalized blocks checked for non-finalized block UTXO spends
* fix(log): reduce verbose logs for block commits (#5348)
* Remove some verbose block write channel logs
* Only warn about tracing endpoint if the address is actually set
* Use CloneError instead of formatting a non-cloneable error
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Increase block verify timeout
* Work around a known block timeout bug by using a shorter timeout
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add RPC timing to zcash-rpc-diff
* Use transaction hash index for verbose block requests, rather than block data
* check if we are at tip for lightwallet wallet tests
* move function
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Combine the lightwalletd sync and gRPC test APIs
* Rewrite the gRPC and full sync tests for the new APIs
* Make zebra_rpc_address optional because only some tests need it
* Check for the zebrad RPC port to open in the right place
* Do the quick lightwalletd integration tests first in the sequential test function
* Ignore the lightwalletd cached state env var in tests that don't want it
* Don't replace the state path in RPC tests
* Enable IO (and timers) on the tip check tokio runtime
* Stop waiting for sync if either waiter thread errors or panics
* Try to speed up slow lightwalletd full syncs
* Don't wait for the tip in send transaction tests, and try to speed up full lightwalletd syncs
* Remove redundant is_lightwalletd_finished store
Co-authored-by: Arya <aryasolhi@gmail.com>
* Fix unused variable error
* Actually create the lightwalletd cached state
* Fix lwd cache check logic
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a new block commit task and channels, that don't do anything yet
* Add last_block_hash_sent to the state service, to avoid database accesses
* Update last_block_hash_sent regardless of commit errors
* Rename a field to StateService.max_queued_finalized_height
* Commit finalized blocks to the state in a separate task
* Check for panics in the block write task
* Wait for the block commit task in tests, and check for errors
* Always run a proptest that sleeps once
* Add extra debugging to state shutdowns
* Work around a RocksDB shutdown bug
* Close the finalized block channel when we're finished with it
* Only reset state queue once per error
* Update some TODOs
* Add a module doc comment
* Drop channels and check for closed channels in the block commit task
* Close state channels and tasks on drop
* Remove some duplicate fields across StateService and ReadStateService
* Try tweaking the shutdown steps
* Update and clarify some comments
* Clarify another comment
* Don't try to cancel RocksDB background work on drop
* Fix up some comments
* Remove some duplicate code
* Remove redundant workarounds for shutdown issues
* Remode a redundant channel close in the block commit task
* Remove a mistaken `!force` shutdown condition
* Remove duplicate force-shutdown code and explain it better
* Improve RPC error logging
* Wait for chain tip updates in the RPC tests
* Wait 2 seconds for chain tip updates before skipping them
* Remove an unnecessary block_in_place()
* Fix some test error messages that were changed by earlier fixes
* Expand some comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
* Actually drop children of failed blocks
* Explain why we drop descendants of failed blocks
* Clarify a comment
* Wait for chain tip updates in a failing test on macOS
* Clean duplicate finalized blocks when the non-finalized state activates
* Send an error when receiving a duplicate finalized block
* Update checkpoint block behaviour, document its consensus rule
* Wait for chain tip changes in inbound_block_height_lookahead_limit test
* Wait for the genesis block to commit in the fake peer set mempool tests
* Disable unreliable mempool verification check in the send transaction test
* Appease rustfmt
* Use clear_finalized_block_queue() everywhere that blocks are dropped
* Document how Finalized and NonFinalized clones are different
* Use the same check as commit_finalized() for finalized block heights
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* use spawn_blocking to run zebra_state::init from start cmd
* uses zebra_state::spawn_init in copy-state command
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move version into a ConnectionInfo struct
* Add negotiated version to ConnectionInfo
Part of this change was generated using:
```
fastmod --fixed-strings ".version(" ".remote_version(" zebra-network
```
* Add the peer address to ConnectionInfo, add ConnectionInfo to Connection
* Return a Client instance from connect_isolated_* functions
This allows library users to access client ConnectionInfo.
* Add and improve debug formatting
* Add peer services and user agent to ConnectionInfo
* Export the Client type, and fix up a zebrad test
* Export types used by the public API
* Split VersionMessage into its own struct
* Use VersionMessage in ConnectionInfo
* Add a public API test for ConnectionInfo
* Wrap ConnectionInfo in an Arc
* Fix some doc links
* Increase the full sync timeout to 28 hours
And put the timeout constant with the rest of them.
* Update developer documentation for the new constant location
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* upgrade config files test
* update cache data to generic in config files
* replace config file v14 with v15
* add test to check if we have the current config stored
* Explain how to fix the test failure
Co-authored-by: teor <teor@riseup.net>
* adds non-blocking writer for tracing subscriber
* use non_blocking writer for the fmt::Layer with the tokio-console feature as well
* adds doc comment to _guard field
* adds acceptance test
* update filter_handle type to use NonBlocking
* adds more detail on lossy non-blocking writer and sets tracing.filter to "trace" in acceptance test
* drops ZebradApp before process::exit(1) in the event of a FrameworkError
* reduces buffered lines limit to 8000
* adds tracing.buffer_limit config and some comments
* update acceptance.rs
* fix acceptance test
* fixes ambigious phrasing in comment
* updates zebrad/src/application.rs
* Find out what the join error is in the GitHub runner tests
* updates acceptance test to use recv_timeout instead of always waiting 10 seconds, removes unnecessary echo command, and reduces # of rpc requests to 500
* see if sleeping for a few seconds before exiting helps the macOS test pass
* Expand exit sleep docs
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* bump prost, tonic and tonic-build
* add protoc as a dependency step in the CI
* bump console-subscriber
* add protoc to `build-crates-individually`
* add protoc to docs build
* install protoc in lint.yml
* change protoc installation location in lint.yml
* add protoc to `Check Cargo.lock is up to date`
* ci(build): keep protoc pinned to the same major version
* ci(build): avoid rate limiting with `arduino/setup-protoc@v1`
* cargo upgrade --workspace console-subscriber
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Co-authored-by: teor <teor@riseup.net>
* add grpc mempool test research
* add a config flag for mempool injection of transactions in test
* Only copy the inner state directory in the send transactions test
* Preload Zcash parameters in some transaction verification tests
* Add a block and transaction Hash method to convert from display order bytes
* Update test coverage docs
* Add debugging output for mempool transaction verification
* Test fetching sent mempool transactions using gRPC
* Add extra log checks to the send transaction test
* Wait for zebrad mempool activation before running gRPC tests
* Update send transaction test for lightwalletd not returning mempool transactions
* Check zebrad logs instead of disabled lightwalletd gRPCs
* Add a debug option that makes RPCs pretend the sync is finished
* Remove an unused debug option
* Remove unused test code and downgrade some logs
* Fix test log checks
* Fix some rustdoc warnings
* Fix a compilation error due to new function arguments
* Make zebrad sync timeouts consistent and remove outdated code
* Document how to increase temporary directory space for tests
* Stop checking for a log that doesn't always happen
* Remove some commented-out code
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Update a comment about run time
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Add new config to new tests from the `main` branch
* Add transactions to the list, rather than replacing the list with each new block
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Add a config for increasing the number of RPC threads
* Add unit tests for parallel RPC servers
* Refactor tests to remove duplicate code
* Update the README
* Actually use parallel threads in some RPC tests
* Rename a function to prepare_block_header_and_transaction_data_batch()
* Fix formatting of test command timeouts and child process output
* Put some #[cfg()]s in the standard Rust location
* Update some test timings
* Allow code timers to be ignored
* adds start as default subcommand for zebrad
* moves EntryPoint to submodule and adds a test
* moves all start tests to config_test to avoid listener conflicts
* Update zebrad/src/application/entry_point.rs docs
* Revert "moves all start tests to config_test to avoid listener conflicts"
This reverts commit 61ce46f5a1.
* Update based on test API changes from another PR
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Handle test failure regexes using Result::Err, rather than panicking
* Add output logs to test context, and add tests for that
* Let empty test child logs be read again (and produce empty output)
* Ignore missing test children when killing with ignore_exited
* Fix a clippy lint
* Rename `line` to `line_result` for clarity
* Revert a redundant context_from() on kill()
* Only ignore "no such process" kill() errors in sync_until() tests
* Log the command timeout when an acceptance test fails
* fix clippy
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Show the arguments of acceptance test functions in the logs
* Show all the logs in the "Run tests" jobs
* Document expected "broken pipe" error from `tee`
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Increase full sync timeout to 24 hours
Expected sync time is ~21 hours as of August 2022.
* Split final checkpoint job into two smaller jobs to avoid timeouts
Also make regexes easier to read.
* Fix a job name typo
* update timeout
* update the doc comment
* Increase test timeouts for Zebra update syncs
* Stop failing the 1740k job if the cached state is after block 1740k
Co-authored-by: teor <teor@riseup.net>
* truncate the number of transactions in send transaction test
* Limit send transaction test to RPC queue length
Co-authored-by: teor <teor@riseup.net>
* update zebra to 1.0.0-beta.13
* add changelog
* update the release date
* Update changelog for cryptographic batch PRs
* Add `bitvec` to the changelog
* Update CHANGELOG.md
* Add zebra-network fix and tokio version bump
* Recommend that users update their rust compiler
* Update disk and network usage for recent large blocks
* Fix spacing
* Add disk and network usage update to README
* apply size suggestions
Co-authored-by: teor <teor@riseup.net>
* add known performance issues
* update changelog
* change release date
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make code execution time logs shorter
* Do ZK parameter preloads in the lightwalletd tests that need them
* Try to re-launch `lightwalletd` when it hangs during sync tests
* Increase full sync timeout
* Clear the `zebrad` logs during `lightwalletd` tests, to avoid logging deadlocks
* Actually clear more than one line of logs
* Check zebrad and lightwalletd output in parallel threads, while waiting for zebrad
* Check zebrad and lightwalletd output in parallel threads, while waiting for lightwalletd
* Improve test logging
* Fix a log typo
* Only wait for lightwalletd once, because its logs stop after the initial sync
* Look for cached state disks for this commit and branch first
* Only copy the state once in the send transactions test
* Wait longer for lightwalletd gRPC server startup
* Add some function docs
* cargo fmt --all
* Fix clippy::let_and_return
* Increase lightwalletd test timeouts for zebrad slowness
* Add a `zebrad_update_sync()` test, that update syncs Zebra without lightwalletd
* Run the zebrad-update-sync test in CI
* Add extra zebrad time to workaround lightwalletd bugs
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography
* Improve anchor validation debugging and error messages
* Work around a test data bug, and save some CPU
* Remove redundant checks for empty shielded data
* Skip generating unused interstitial treestates
* Do disk fetches and quick checks, then CPU-heavy cryptography
* Wrap HistoryTree in an Arc in the state
* Run CPU-intensive chain validation and updates in parallel rayon threads
* Refactor to prepare for parallel tree root calculations
* Run finalized state note commitment tree root updates in parallel rayon threads
* Update finalized state note commitment trees using parallel rayon threads
* Fix a comment typo and add a TODO
* Split sprout treestate fetch into its own function
* Move parallel note commitment trees to zebra-chain
* Re-calculate the tree roots in the same parallel batches
* Do non-finalized note commitment tree updates in parallel threads
* Update comments about note commitment tree rebuilds
* Do post-fork tree updates in parallel threads
* Add a TODO for parallel tree updates in tests
* Fix broken intra-doc links
* Clarify documentation for sprout treestates
* Spawn large database reads into blocking tokio threads
* Concurrently read all blocks, headers, and transactions from disk
* Run zebra-state transaction deserialization on a rayon thread
* Update comments about note commitment tree rebuilds
* Add info-level logs to time note commitment tree rebuilds
* Log the number of rebuilt blocks after a fork, and the time per block
* Move humantime formats to zebra-chain
* Use human-friendly time formatting
* Only fetch block headers from the database to answer headers requests
* Move writing to the database to a blocking thread
* Add blocking threads to tests that need them
* Remove mempool downloader requirement for a populated state
And stop populating states that don't need it,
so we can use tokio::time::pause() in those tests.
* Improve debugging for an intermittent test failure
* Try to avoid a race condition populating the mempool in tests
* try to do deserialization of transaction in a rayon thread
* Try tokio::task::block_in_place instead
* fix tests
* add deserialize block into rayon pool
* fill some docs
Co-authored-by: teor <teor@riseup.net>
* Initialize the rayon threadpool with a new config for CPU-bound threads
* Verify proofs and signatures on the rayon thread pool
* Only spawn one concurrent batch per verifier, for now
* Allow tower-batch to queue multiple batches
* Fix up a potentially incorrect comment
* Rename some variables for concurrent batches
* Spawn multiple batches concurrently, without any limits
* Simplify batch worker loop using OptionFuture
* Clear pending batches once they finish
* Stop accepting new items when we're at the concurrent batch limit
* Fail queued requests on drop
* Move pending_items and the batch timer into the worker struct
* Add worker fields to batch trace logs
* Run docker tests on PR series
* During full verification, process 20 blocks concurrently
* Remove an outdated comment about yielding to other tasks
* Return the maximum checkpoint height from the chain verifier
* Return the verified block height from the sync downloader
* Track the verified height in the syncer
* Use a lower concurrency limit during full verification
* Get the tip from the state before the first verified block
* Limit the number of submitted download and verify blocks in a batch
* Adjust lookahead limits when transitioning to full verification
* Keep unused extra hashes and submit them to the downloader later
* Remove redundant verified_height and state_tip()
* Split the checkpoint and full verify concurrency configs
* Decrease full verification concurrency to 5 blocks
10 concurrent blocks causes 3 minute stalls on some blocks on my machine.
(And it has about 4x as many cores as a standard machine.)
* cargo +stable fmt --all
* Remove a log that's verbose with smaller lookahead limits
* Apply the full verify concurrency limit to the inbound service
* Add a summary of the config changes to the CHANGELOG
* Increase the default full verify concurrency limit to 30
* Checkout zebra in each job to avoid warnings
But put TODOs where we might be able to skip checkouts
* Split log following into sprout checkpoints, sapling/orchard checkpoints, and full validation
* Make job IDs shorter
* Use /dev/stderr because docker doesn't have a tty
* remove pipefail
* Revert "remove pipefail"
This reverts commit a7ee37bebdc107a4215e7dd307b189d925969234.
* Make tee ignore errors writing to a grep pipe
* Avoid launching multiple docker instances for duplicate jobs
* Ignore broken pipe error messages and statuses
* fix(ci): docker wait not finding container
We had this issue before, I can't recall if this was a parsing error between GitHub Actions and gcloud `--command` parsing, but we had to change this into two pieces.
This implementation keeps it how we did it before 9b9578c999/.github/workflows/test.yml (L235-L243)
* docs: remove pending TODO
We can't remove `actions/checkout` nor set `create_credentials_file` to `false` as next steps won't be able to authenticate to GCP.
We can surely remove `actions/checkout` and leave `create_credentials_file` as `true`, but this will raise a warning on each step, and there's no benefit of doing so.
* Show `docker wait` and `gcloud ssh` output
* If `docker wait` fails, get the exit code using `docker inspect`
* Make full sync tests go all the way to the tip
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Update Zebra to 1.0.0-beta.12
* Simplify tower-* versioning by updating both to 0.2.27
* Simplify version search and replace by removing an old version from the docs
* Update Cargo.lock
* Add draft changelog as of PR #4693
* Update CHANGELOG to PR #4680
* Document configs that need compile-time features
* Document features in zebrad/src/lib.rs
* Link to the feature list from the README
* Remove some outdated README info
* Move some detailed README info to the `zebrad` crate docs
dependabot[bot]
teor
Arya
Deirdre Connolly
Alfredo Garcia
Marek
Conrado Gouvea