* Create a `SyncStatus` helper type
Keeps track if the synchronizer is close to the chain tip or not.
* Refactor `ChainSync` ctor. to return `SyncStatus`
Change the constructor API so that it returns a higher level construct.
* Test if `SyncStatus` waits for the chain tip
Test if waiting for the chain tip to be reached correctly finishes when
the chain tip is reached. This is done by sending recent sync lengths to
the `SyncStatus` instance, and checking that every time a separate
`SyncStatus` instance determines it has reached the tip the original
instance wakes up.
* Add a temporary attribute to allow dead code
The code added isn't used yet, so we'll add a temporary waiver until
another PR is merged to use them.
* Always prefer the non-finalized tip in ChainTipSender
This significantly simplifies the internal implementation of ChainTipSender.
Also make the methods and types a bit more generic.
* Update ChainTipSender with blocks, not heights
Also fix a bug where queued non-finalized blocks would clear the chain tip.
* Provide a best tip hash in ChainTip receivers
* Skip finalized blocks once the non-finalized state is active
* Add tip hash and NoChainTip tests
* Remove a redundant finalized tip update
* Skip `None` updates to the finalized tip
The finalized and non-finalized tips never update to `None`
once they have added at least one block.
* Stop committing finalized queued blocks if there is an error
Also return the highest committed queued block.
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
This avoids peer set contention when most peers are busy.
Also exit the task if the peer service returns a readiness error,
because that means it's permanently unusable.
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* First pass at a Mempool Service, incl. a storage layer underneath
* Fixed up Mempool service and storage
* allow dead code where needed
* clippy
* typo
* only drain if the mempool is full
* add a basic storage test
* remove space
* fix test for when MEMPOOL_SIZE change
* group some imports
* add a basic mempool service test
* add clippy suggestions
* remove not needed allow dead code
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Rename BestTipHeight so it can be generalised to ChainTipSender
`fastmod BestTipHeight ChainTipSender zebra*`
For senders:
`fastmod best_tip_height chain_tip_sender zebra*`
For receivers:
`fastmod best_tip_height chain_tip_receiver zebra*`
* Rename best_tip_height module to chain_tip
* Wrap the chain tip watch channel in a ChainTipReceiver type
* Create a ChainTip trait to avoid tricky crate dependencies
And add convenience impls for optional and empty chain tips.
* Use the ChainTip trait in zebra-network
* Replace `Option<ChainTip>` with `NoChainTip`
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add `Arc<Transaction>` conversions for Transaction IDs
* Use UnminedTxId as the transaction verifier response type
* Use UnminedTx in transaction verifier mempool requests
* Refactor is_mempool into a transaction verifier request method
* Order derives in alphabetical order
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Return a transaction verifier from `zebra_consensus::init`
This verifier is temporarily created separately from the block verifier's
transaction verifier.
* Return the same transaction verifier used by the block verifier
* Clarify that the mempool verifier is the transaction verifier
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* add value balances to non finalized state
* fix 2 tests
* fix remaining constrain issues in tests
* extend value pool test to non finalized
* WIP: fix tests after adding value pools to non-finalized state (#2647)
* Update Chain::eq_internal_state with Chain.value_balance
Also increase the number of cases in its tests,
because they didn't detect this bug.
* Calculate the chain value pool change before `Chain::push`
Code
- store the chain value pool change in `ContextuallyValidBlock`
- convert `PreparedBlock` to `ContextuallyValidBlock` using `with_block_and_spent_utxos`
(rather than `from` or `into`)
- replace `block_utxos` with `new_outputs` in `PreparedBlock`
- replace `block_utxos` with `chain_value_pool_change` in `ContextuallyValidBlock`
Tests
- create test methods for `PreparedBlock` and `ContextuallyValidBlock`
- use `test_with_zero_chain_pool_change` or `test_with_zero_spent_utxos`
to make tests pass
* fix conflicts
* build `set_current_value_pool()` only for tests
* remove redundant cfgs
* change cfg of set_current_value_pool()
* Clarify some chain field documentation
* Fix bugs in the non-finalized chain value pool calculations
1. Only revert the chain value pool balances when the tip is popped.
Don't modify them when the root is finalized.
2. Only update or revert the chain value pool balances once per block.
(Previously, the block changes were multiplied by the number of *transactions*.)
And make corresponding changes to method names and documentation.
* Add extra proptests to try to identify value balance failures
* Simplify some transaction generation code
* Add extra debugging info to value balance errors
* Actually update non-finalized chain value pools in `UpdateWith`
Previously, we were dropping the updated value pools in the `Ok` result.
So the initial (finalized) chain value pool balances were never modified.
* Rename and document value balance add methods
The names and documentation of these methods were confusing.
* Create genesis-based proptests that check chain value pools
* Increase coverage for some test vectors
* Test each chain value balance calculation for blocks 0-10
* Make continuous blockchain test errors easier to debug
* Test the exact transparent pool values for the first few blocks
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
`Message::Inv(TxId+)` is a transaction advertisement,
so it should be converted into `Request::AdvertiseTransactionIds`.
This is a copy-paste mistake from the original zebra-network
implementation.
* Create initial `mempool::Crawler` type
The mempool crawler is responsible for periodically asking peers for
transactions to insert into the local mempool. This initial
implementation will periodically ask for transactions, but won't do
anything with them yet.
Also, the crawler is currently configured to be always enabled, but this
should be fixed to avoid crawling while Zebra is still syncing the
chain.
* Add a timeout to peer responses
Prevent the crawler from getting stuck if there's communication with a
peer that takes too long to respond.
* Run the mempool crawler in Zebra
Spawn a task for the crawler when Zebra starts.
* Test if the crawler is sending requests
Create a mock for the `PeerSet` service to intercept requests and verify
that the transaction requests are sent periodically.
* Use `full` Tokio features when testing
Make it simpler to select the features for test builds.
Co-authored-by: teor <teor@riseup.net>
* Link to the issue for crawler activation
Make it easy to navigate from the `TODO` comment to the current project
planning.
Co-authored-by: teor <teor@riseup.net>
* Link to the issue for downloading transactions
Make it easy to navigate from the `TODO` comment to the current project
planning.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
* Add validation of ZIP-221 and ZIP-244 commitments
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Add auth commitment check in the finalized state
* Reset the verifier when comitting to state fails
* Add explanation comment
* Add test with fake activation heights
* Add generate_valid_commitments flag
* Enable fake activation heights using env var instead of feature
* Also update initial_tip_hash; refactor into progress_from_tip()
* Improve comments
* Add fake activation heights test to CI
* Fix bug that caused commitment trees to not match when generating partial arbitrary chains
* Add ChainHistoryBlockTxAuthCommitmentHash::from_commitments to organize and deduplicate code
* Remove stale comment, improve readability
* Allow overriding with PROPTEST_CASES
* partial_chain_strategy(): don't update note commitment trees when not needed; add comment
Co-authored-by: teor <teor@riseup.net>
* Minimal recent sync lengths implementation
Also includes metrics and logging, to make diagnosing bugs easier.
* Add logging to check what happens when Zebra reaches the chain tip
* Add tests for recent sync lengths
- initially empty
- pruned to correct length
- newest entries go first
* Drop a redundant `/` from a Cargo.lock URL
This seems to be a nightly or beta Rust change,
but hopefully stable just accepts it.
* Use metrics histograms to avoid overwriting values
* Add detailed syncer monitoring dashboard
* Increase the recent sync length to 4
This length makes it easier to distinguish between temporary and
sustained errors/syncs.
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Improve error logging in rejection_restores_internal_state
* Summarise history tree peaks in debug output
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* add value pools to the database
* remove redundant genesis block check
* use update_with_chain_value_pool_change()
* remove constrains
* remove height from the database
* remove calls to chain_value_pool_change
* clippy
* use the "correct" value balances
* bump the database format
* remove everything that is not finalized state
* clippy
* rustfmt
* use all spent utxos
* add new_outputs utxos to all_utxos_spent_by_block
* remove panic
* add finalized state value pool test
* clippy
* clippy 2
* move import
* fix import
* rustfmt
Co-authored-by: teor <teor@riseup.net>
* Rename internal network requests for wide transaction IDs
fastmod TransactionsByHash TransactionsById zebra*
fastmod AdvertiseTransactions AdvertiseTransactionIds zebra*
fastmod MempoolTransactions MempoolTransactionIds zebra*
fastmod TransactionHashes TransactionIds zebra*
* Update network transaction request/response comments
* Rename a transaction hash method for wide transaction IDs
fastmod transaction_hashes transaction_ids zebra-network
* Add UnminedTxId methods and conversions for InventoryHash
* Map WtxIds to unmined transaction network messages
Also, use UnminedTxId and UnminedTx in:
* Zebra's internal request and response format, and
* external Zcash network protocol messages.
* Enable WtxId mempool inventory tracking for peers
* Further clarify transaction IDs
* Use Witnessed rather than Wide for transaction IDs
And rename narrow to legacy when it only applies to v1-v4 transactions.
Otherwise, rename it to mined ID.
* Rename a missed binding
* Remove an incorrectly named binding
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Only use -t flag to docker run, set SSH keep alive
* Remove SSH flag for now
* Add ssh flag back to test.yml gcloud compute ssh command
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Make the `AuthDigest` display order match transaction IDs
And derive `Hash`, just like transaction IDs.
Don't derive `serde` for now, because it's not needed.
* Move transaction::Hash test to tests module
* Add a simple AuthDigest display order test
* Add a WtxId type for wide transaction IDs
* Add conversions between transaction IDs and bytes
* Use the WtxId type in external network protocol messages
* Generate chains with valid chain value pool balances
* Move MAX_PARTIAL_CHAIN_BLOCKS to zebra-chain
* Fix generated value overflow based on the maximum number of values
And split it into its own method.
* Split fix_remaining_value into smaller methods
* Remove unused methods
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
The zebra-state integration test compiled successfully during workspace
builds.
But it couldn't find the `init_test` function when the crate was built
by itself. This happens because the function is only active for `test`
and `proptest-impl` builds.
This commit removes the dependency on that function.
* Refactor HistoryTree into NonEmptyHistoryTree and HistoryTree
* HistoryTree: use Deref instead of AsRef; remove unneeded PartialEq
* ZIP-221: Validate chain history commitments in the non-finalized state (#2301)
* sketch of implementation
* refined implementation; still incomplete
* update librustzcash, change zcash_history to work with it
* simplified code per review; renamed MMR to HistoryTree
* expand HistoryTree implementation
* handle and propagate errors
* simplify check.rs tracing
* add suggested TODO
* add HistoryTree::prune
* fix bug in pruning
* fix compilation of tests; still need to make them pass
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* improvements from code review
* improve check.rs comments and variable names
* fix HistoryTree which should use BTreeMap and not HashMap; fix non_finalized_state prop tests
* fix finalized_state proptest
* fix non_finalized_state tests by setting the correct commitments
* renamed mmr.rs to history_tree.rs
* Add HistoryTree struct
* expand non_finalized_state protest
* fix typo
* Add HistoryTree struct
* Update zebra-chain/src/primitives/zcash_history.rs
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* fix formatting
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* history_tree.rs: fixes from code review
* fixes to work with updated HistoryTree
* Improvements from code review
* Add Debug implementations to allow comparing Chains with proptest_assert_eq
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Improvements from code review
* Restore blocks returned by PreparedChain since other tests broken; adjust tests with history trees
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Make Amount arithmetic more generic
To modify generated amounts, we need some extra operations on `Amount`.
We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.
* Add a constrain method for ValueBalance
* Derive Eq for ValueBalance
* impl Neg for ValueBalance
* Make some Amount arithmetic expectations explicit
* Explain why we use i128 for multiplication
And expand the overflow error details.
* Expand Amount::sum error details
* Make amount::Error field order consistent
* Rename an amount::Error variant to Constraint, so it's clearer
* Add specific pool variants to ValueBalanceError
* Update coinbase remaining value consensus rule comment
This consensus rule was updated recently to include coinbase transactions,
but Zebra doesn't check block subsidy or miner fees yet.
* Add test methods for modifying transparent values and shielded value balances
* Temporarily set values and value balances to zero in proptests
In both generated chains and proptests that construct their own transactions.
Using zero values reduces value calculation and value check test coverage.
A future change will use non-zero values, and fix them so the check passes.
* Add extra fields to remaining transaction value errors
* Swap the transparent value balance sign to match shielded value balances
This makes the signs of all the chain value pools consistent.
* Use a NonNegative constraint for transparent values
This fix:
* makes the type signature match the consensus rules
* avoids having to write code to handle negative values
* Allocate total generated transaction input value to outputs
If there isn't enough input value for an output, set it to zero.
Temporarily reduce all generated values to avoid overflow.
(We'll remove this workaround when we calculate chain value balances.)
* Consistently use ValueBalanceError for ValueBalances
* Make the value balance signs match the spec
And rename and document methods so their signs are clearer.
* Convert amount::Errors to specific pool ValueBalanceErrors
* Move some error changes to the next PR
* Add extra info to remaining transaction value errors (#2585)
* Distinguish between overflow and negative remaining transaction value errors
And make some error types cloneable.
* Add methods for updating chain value pools (#2586)
* Move amount::test to amount::tests:vectors
* Make ValueBalance traits more consistent with Amount
- implement Add and Sub variants with Result and Assign
- derive Hash
* Clarify some comments and expects
* Create ValueBalance update methods for blocks and transactions
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Make Amount arithmetic more generic
To modify generated amounts, we need some extra operations on `Amount`.
We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.
* Add a constrain method for ValueBalance
* Derive Eq for ValueBalance
* impl Neg for ValueBalance
* Make some Amount arithmetic expectations explicit
* Explain why we use i128 for multiplication
And expand the overflow error details.
* Expand Amount::sum error details
* Make amount::Error field order consistent
* Rename an amount::Error variant to Constraint, so it's clearer
* Add specific pool variants to ValueBalanceError
* add serialization for value balances
* change test names
* change panic messages
* add a deserialization test
* return the errors from `from_bytes()` methods
* add prop test for serialize/deserialize Amount
* Move amount proptests to amount::tests::prop
Co-authored-by: teor <teor@riseup.net>
* Simplify state service initialization in test
Use the test helper function to remove redundant code.
* Create `BestTipHeight` helper type
This type abstracts away the calculation of the best tip height based on
the finalized block height and the best non-finalized chain's tip.
* Add `best_tip_height` field to `StateService`
The receiver endpoint is currently ignored.
* Return receiver endpoint from service constructor
Make it available so that the best tip height can be watched.
* Update finalized height after finalizing blocks
After blocks from the queue are finalized and committed to disk, update
the finalized block height.
* Update best non-finalized height after validation
Update the value of the best non-finalized chain tip block height after
a new block is committed to the non-finalized state.
* Update finalized height after loading from disk
When `FinalizedState` is first created, it loads the state from
persistent storage, and the finalized tip height is updated. Therefore,
the `best_tip_height` must be notified of the initial value.
* Update the finalized height on checkpoint commit
When a checkpointed block is commited, it bypasses the non-finalized
state, so there's an extra place where the finalized height has to be
updated.
* Add `best_tip_height` to `Handshake` service
It can be configured using the `Builder::with_best_tip_height`. It's
currently not used, but it will be used to determine if a connection to
a remote peer should be rejected or not based on that peer's protocol
version.
* Require best tip height to init. `zebra_network`
Without it the handshake service can't properly enforce the minimum
network protocol version from peers. Zebrad obtains the best tip height
endpoint from `zebra_state`, and the test vectors simply use a dummy
endpoint that's fixed at the genesis height.
* Pass `best_tip_height` to proto. ver. negotiation
The protocol version negotiation code will reject connections to peers
if they are using an old protocol version. An old version is determined
based on the current known best chain tip height.
* Handle an optional height in `Version`
Fallback to the genesis height in `None` is specified.
* Reject connections to peers on old proto. versions
Avoid connecting to peers that are on protocol versions that don't
recognize a network update.
* Document why peers on old versions are rejected
Describe why it's a security issue above the check.
* Test if `BestTipHeight` starts with `None`
Check if initially there is no best tip height.
* Test if best tip height is max. of latest values
After applying a list of random updates where each one either sets the
finalized height or the non-finalized height, check that the best tip
height is the maximum of the most recently set finalized height and the
most recently set non-finalized height.
* Add `queue_and_commit_finalized` method
A small refactor to make testing easier. The handling of requests for
committing non-finalized and finalized blocks is now more consistent.
* Add `assert_block_can_be_validated` helper
Refactor to move into a separate method some assertions that are done
before a block is validated. This is to allow moving these assertions
more easily to simplify testing.
* Remove redundant PoW block assertion
It's also checked in
`zebra_state::service::check::block_is_contextually_valid`, and it was
getting in the way of tests that received a gossiped block before
finalizing enough blocks.
* Create a test strategy for test vector chain
Splits a chain loaded from the test vectors in two parts, containing the
blocks to finalize and the blocks to keep in the non-finalized state.
* Test committing blocks update best tip height
Create a mock blockchain state, with a chain of finalized blocks and a
chain of non-finalized blocks. Commit all the blocks appropriately, and
verify that the best tip height is updated.
Co-authored-by: teor <teor@riseup.net>
dependabot[bot]
Janito Vaqueiro Ferreira Filho
Alfredo Garcia
Marek
teor
Deirdre Connolly
Conrado Gouvea
Pili Guerra