* Add minimum queued block height to metrics
* Add metrics to sent block cache
* Add metrics for the last sent finalized height
* Refactor sent metrics
* Add new block channel metrics to a grafana dashboard
* Add a new block commit task and channels, that don't do anything yet
* Add last_block_hash_sent to the state service, to avoid database accesses
* Update last_block_hash_sent regardless of commit errors
* Rename a field to StateService.max_queued_finalized_height
* Commit finalized blocks to the state in a separate task
* Check for panics in the block write task
* Wait for the block commit task in tests, and check for errors
* Always run a proptest that sleeps once
* Add extra debugging to state shutdowns
* Work around a RocksDB shutdown bug
* Close the finalized block channel when we're finished with it
* Only reset state queue once per error
* Update some TODOs
* Add a module doc comment
* Drop channels and check for closed channels in the block commit task
* Close state channels and tasks on drop
* Remove some duplicate fields across StateService and ReadStateService
* Try tweaking the shutdown steps
* Update and clarify some comments
* Clarify another comment
* Don't try to cancel RocksDB background work on drop
* Fix up some comments
* Remove some duplicate code
* Remove redundant workarounds for shutdown issues
* Remode a redundant channel close in the block commit task
* Remove a mistaken `!force` shutdown condition
* Remove duplicate force-shutdown code and explain it better
* Improve RPC error logging
* Wait for chain tip updates in the RPC tests
* Wait 2 seconds for chain tip updates before skipping them
* Remove an unnecessary block_in_place()
* Fix some test error messages that were changed by earlier fixes
* Expand some comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
* Actually drop children of failed blocks
* Explain why we drop descendants of failed blocks
* Clarify a comment
* Wait for chain tip updates in a failing test on macOS
* Clean duplicate finalized blocks when the non-finalized state activates
* Send an error when receiving a duplicate finalized block
* Update checkpoint block behaviour, document its consensus rule
* Wait for chain tip changes in inbound_block_height_lookahead_limit test
* Wait for the genesis block to commit in the fake peer set mempool tests
* Disable unreliable mempool verification check in the send transaction test
* Appease rustfmt
* Use clear_finalized_block_queue() everywhere that blocks are dropped
* Document how Finalized and NonFinalized clones are different
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* updates comments, renames send_process_queued, other minor cleanup
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* updates comments, renames send_process_queued, other minor cleanup
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* removes duplicate field definitions on StateService that were a result of a bad merge
* update NotReadyToBeCommitted error message
* Appear rustfmt
* Fix doc links
* Rename a function to initial_contextual_validity()
* Do error tasks on Err, and success tasks on Ok
* Simplify parent_error_map truncation
* Rewrite best_tip() to use tip()
* Rename latest_mem() to latest_non_finalized_state()
```sh
fastmod latest_mem latest_non_finalized_state zebra*
cargo fmt --all
```
* Simplify latest_non_finalized_state() using a new WatchReceiver API
* Expand some error messages
* Send the result after updating the channels, and document why
* wait for chain_tip_update before cancelling download in mempool_cancel_mined
* adds `sent_non_finalized_block_hashes` field to StateService
* adds batched sent_hash insertions and checks sent hashes in queue_and_commit_non_finalized before adding a block to the queue
* check that the `curr_buf` in SentHashes is not empty before pushing it to the `sent_bufs`
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix rustfmt
* Check for finalized block heights using zs_contains()
* adds known_utxos field to SentHashes
* updates comment on SentHashes.add method
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* return early when there's a duplicate hash in QueuedBlocks.queue instead of panicking
* Make finalized UTXOs near the final checkpoint available for full block verification
* Replace a checkpoint height literal with the actual config
* Update mainnet and testnet checkpoints - 7 October 2022
* Fix some state service init arguments
* Allow more lookahead in the downloader, but less lookahead in the syncer
* Add the latest config to the tests, and fix the latest config check
* Increase the number of finalized blocks checked for non-finalized block UTXO spends
* fix(log): reduce verbose logs for block commits (#5348)
* Remove some verbose block write channel logs
* Only warn about tracing endpoint if the address is actually set
* Use CloneError instead of formatting a non-cloneable error
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Increase block verify timeout
* Work around a known block timeout bug by using a shorter timeout
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a new block commit task and channels, that don't do anything yet
* Add last_block_hash_sent to the state service, to avoid database accesses
* Update last_block_hash_sent regardless of commit errors
* Rename a field to StateService.max_queued_finalized_height
* Commit finalized blocks to the state in a separate task
* Check for panics in the block write task
* Wait for the block commit task in tests, and check for errors
* Always run a proptest that sleeps once
* Add extra debugging to state shutdowns
* Work around a RocksDB shutdown bug
* Close the finalized block channel when we're finished with it
* Only reset state queue once per error
* Update some TODOs
* Add a module doc comment
* Drop channels and check for closed channels in the block commit task
* Close state channels and tasks on drop
* Remove some duplicate fields across StateService and ReadStateService
* Try tweaking the shutdown steps
* Update and clarify some comments
* Clarify another comment
* Don't try to cancel RocksDB background work on drop
* Fix up some comments
* Remove some duplicate code
* Remove redundant workarounds for shutdown issues
* Remode a redundant channel close in the block commit task
* Remove a mistaken `!force` shutdown condition
* Remove duplicate force-shutdown code and explain it better
* Improve RPC error logging
* Wait for chain tip updates in the RPC tests
* Wait 2 seconds for chain tip updates before skipping them
* Remove an unnecessary block_in_place()
* Fix some test error messages that were changed by earlier fixes
* Expand some comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
* Actually drop children of failed blocks
* Explain why we drop descendants of failed blocks
* Clarify a comment
* Wait for chain tip updates in a failing test on macOS
* Clean duplicate finalized blocks when the non-finalized state activates
* Send an error when receiving a duplicate finalized block
* Update checkpoint block behaviour, document its consensus rule
* Wait for chain tip changes in inbound_block_height_lookahead_limit test
* Wait for the genesis block to commit in the fake peer set mempool tests
* Disable unreliable mempool verification check in the send transaction test
* Appease rustfmt
* Use clear_finalized_block_queue() everywhere that blocks are dropped
* Document how Finalized and NonFinalized clones are different
* Use the same check as commit_finalized() for finalized block heights
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* use spawn_blocking to run zebra_state::init from start cmd
* uses zebra_state::spawn_init in copy-state command
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Look back 10,000 blocks on testnet for a legacy chain
* Use a smaller number of legacy chain blocks in the tests
* Fix test assertion error message
* Clarify legacy chain check comment
* cargo fmt --all
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move AwaitUtxos next to the other shared writeable state requests
* Rename ReadResponse::Utxos to ReadResponse::AddressUtxos
```sh
fastmod Utxos AddressUtxos zebra*
```
* Rename an out_point variable to outpoint for consistency
* Rename transparent_utxos to address_utxos
```sh
fastmod transparent_utxos address_utxos zebra*
```
* Run AwaitUtxo without accessing shared mutable chain state
* Fix some incorrect comments
* Explain why some concurrent reads are ok
* Add a TODO
* Stop using self.mem in AwaitUtxo requests
* Update state service module documentation
* Move the QueuedBlock type into the queued_blocks module
* Explain how spent UTXOs are treated by the state
* Clarify how cached Chains impact state read requests
And move repeated comments to the module header.
* fastmod ChainUtxo BestChainUtxo zebra*
* Add an AnyChainUtxo request
* Make AwaitUtxo non-blocking
* Move the finalized block queue into the StateService
* Move the queued_blocks module to the state service
* Move QueuedFinalized into queued_blocks
* Move the queued_blocks tests into their own module
* Make the FinalizedState cloneable
* cleanup of repetitive code
* fixes merge by adding back req.count_metric and removing the metrics::counter in AwaitUtxo
Co-authored-by: teor <teor@riseup.net>
* Move the finalized block queue into the StateService
* Move the queued_blocks module to the state service
* Move QueuedFinalized into queued_blocks
* Move the queued_blocks tests into their own module
* Make the FinalizedState cloneable
* Move AwaitUtxos next to the other shared writeable state requests
* Rename ReadResponse::Utxos to ReadResponse::AddressUtxos
```sh
fastmod Utxos AddressUtxos zebra*
```
* Rename an out_point variable to outpoint for consistency
* Rename transparent_utxos to address_utxos
```sh
fastmod transparent_utxos address_utxos zebra*
```
* Run AwaitUtxo without accessing shared mutable chain state
* Fix some incorrect comments
* Explain why some concurrent reads are ok
* Add a TODO
* Stop using self.mem in AwaitUtxo requests
* Update state service module documentation
* Move the QueuedBlock type into the queued_blocks module
* Explain how spent UTXOs are treated by the state
* Clarify how cached Chains impact state read requests
And move repeated comments to the module header.
* fastmod ChainUtxo BestChainUtxo zebra*
* Add an AnyChainUtxo request
* Make AwaitUtxo non-blocking
* Add TODOs for finalized blocks, non-finalized blocks, and reads
* Document how state requests use shared state
* Add a ReadStateService to the StateService
And cleanup service struct fields.
* Redirect Block and Transaction Requests to the ReadStateService
* Put AddressBalance in a consistent enum order
* Turn a repeated comment into documentation
* Tidy doc links
* Run Tip requests concurrently
* Remove some redundant timers
* Run Depth requests concurrently
* Run BlockLocator requests concurrently
* Move BlockLocator tests
* Run FindBlockHashes requests concurrently
* Run FindBlockHeaders requests concurrently
* Use a constant in documentation
Co-authored-by: Marek <mail@marek.onl>
* Link that constant correctly
* Expand block_locator() documentation
* Clarify the difference between tower::Buffers and the state's ordered queues
* Explain block locators better
Co-authored-by: Marek <mail@marek.onl>
* Add history trees for each height in non-fin state
* Refactor formatting
* Pass the treestate to the finalized state
I created a new structure `FinalizedBlockWithTrees` that wraps the
treestate and the finalized block. I did that because the original
`FinalizedBlock` is `Eq`, but `HistoryTree` can't be `Eq`.
This makes Zebra faster because:
1. The finalized state doesn't retrieve the treestate from the disk if
the non-finalized state supplies it.
2.The finalized state doesn't recompute the treestate if the
non-finalized state supplies it.
* Check block commitment before updating hist tree
* Store Sprout commitment trees in non-fin state
* Send trees for the root block to fin-state
When committing a block and sending the treestate from the non-finalized
state to the finalized state, Zebra was sending trees that correspond to
the tip block instead of trees that correspond to the root block of the
best chain. This commit fixes that.
* Refactor doc comments
* Refactor block finalization
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* cargo upgrade --workspace rocksdb
* Add a deny.toml exception for bindgen 0.59
* Move `valid()` and `status()` methods to raw iterators
* Update some outdated comments
* Panic on iterator failures
We might want to change this if there are any common failure modes.
* allow(clippy::unwrap_in_result) in some methods
* Fix clippy::let_and_return
* Increase lightwalletd test timeouts for zebrad slowness
* Add a `zebrad_update_sync()` test, that update syncs Zebra without lightwalletd
* Run the zebrad-update-sync test in CI
* Add extra zebrad time to workaround lightwalletd bugs
* Add a CodeTimer diagnostic struct for long-running code
* Time state init and each state request, log when it takes too long
* Add code locations to execution timers
* Instrument state futures and functions with tracing spans
* Only log each code timer once
* Make displayed times shorter
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography
* Improve anchor validation debugging and error messages
* Work around a test data bug, and save some CPU
* Remove redundant checks for empty shielded data
* Skip generating unused interstitial treestates
* Do disk fetches and quick checks, then CPU-heavy cryptography
* Wrap HistoryTree in an Arc in the state
* Run CPU-intensive chain validation and updates in parallel rayon threads
* Refactor to prepare for parallel tree root calculations
* Run finalized state note commitment tree root updates in parallel rayon threads
* Update finalized state note commitment trees using parallel rayon threads
* Fix a comment typo and add a TODO
* Split sprout treestate fetch into its own function
* Move parallel note commitment trees to zebra-chain
* Re-calculate the tree roots in the same parallel batches
* Do non-finalized note commitment tree updates in parallel threads
* Update comments about note commitment tree rebuilds
* Do post-fork tree updates in parallel threads
* Add a TODO for parallel tree updates in tests
* Fix broken intra-doc links
* Clarify documentation for sprout treestates
* Spawn large database reads into blocking tokio threads
* Concurrently read all blocks, headers, and transactions from disk
* Run zebra-state transaction deserialization on a rayon thread
* Only fetch block headers from the database to answer headers requests
* Move writing to the database to a blocking thread
* Add blocking threads to tests that need them
* Remove mempool downloader requirement for a populated state
And stop populating states that don't need it,
so we can use tokio::time::pause() in those tests.
* Improve debugging for an intermittent test failure
* Try to avoid a race condition populating the mempool in tests
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fa.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
* Fix some warnings produced by `cargo doc`
* Fix some rustdoc warnings
* Fix some warnings
* Refactor some changes
* Fix some rustdoc warnings
* Fix some rustdoc warnings
* Resolve various TODOs
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Upgrade tracing and related dependencies
```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber
color-eyre
tracing-flame
tracing-journald
sentry
sentry-tracing
metrics
metrics-exporter-prometheus
reqwest
```
* Update duplicate dependency checks
* Enable the tracing/env-filter feature
* Fix type inference for metrics
Manual changes, plus:
```sh
fastmod "as _" "as f64"
```
* Tidy up some unrelated test code
* Update metrics-exporter-prometheus API
And make unused dependencies optional.
* Adjust test regexes to new tracing format
Also fix some regex bugs, and refactor to simplify.
* Disable color-eyre span traces and track caller in release builds
* Add a feature that enables extra debugging in release builds
* Clean up some redundant features
* Increase a test timeout
* Impl the elementary structure of the `z_gettreestate` RPC
* Fix merging bugs
* Fix a merge bug
* Fix a merge bug
* Move a derive attribute
Co-authored-by: teor <teor@riseup.net>
* Clarify the support of negative heights
* Add Orchard note commitment trees to the response
* Add the time to the response
* Finalize the `z_gettreestate` RPC
* Add a note that verified blocks have coinbase height
* Refactor `from_str` for `HashOrHeight`
* Fix a mistake in the docs
Co-authored-by: teor <teor@riseup.net>
* Clarify request types
Co-authored-by: teor <teor@riseup.net>
* Simplify `hash_or_height` conversion to height
Co-authored-by: teor <teor@riseup.net>
* Add a TODO about optimization
Co-authored-by: teor <teor@riseup.net>
* Add a doc comment
* Make sure Sapling & Orchard trees don't get mixed up
* Serialize Sapling commitment trees
* Refactor some comments
* Serialize Orchard commitment trees
* Serialize block heights
* Simplify the serialization of commitment trees
* Remove the block time from the RPC response
* Simplify the serialization of block heights
* Put Sapling & Orchard requests together
* Remove a redundant TODO
* Add block times to the RPC response
* Derive `Clone, Debug, Eq, PartialEq` for `GetTreestate`
Co-authored-by: teor <teor@riseup.net>
* Derive `Clone`, `Debug`, `Eq` and `PartialEq` for `SerializedTree`
* Document the fields of `GetTreestate`
* Skip the serialization of empty trees
This ensures compatibility with `zcashd` in the `z_gettreestate` RPC.
* Document the `impl` of `merkle_tree::Hashable` for nodes
* Make the structure of the JSON response consistent with `zcashd`
* Derive `Eq` for nodes
Co-authored-by: teor <teor@riseup.net>
* Convert Sapling commitment trees to a format compatible with zcashd
* Refactor the conversion of Sapling commitment trees
* Refactor some comments
* Refactor comments
* Add a description of the conversion
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Fix comment indenting
* Document the conversion between the dense and sparse formats
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* implement display for `Script`
* implement `getaddressutxos`
* fix space
* normalize list of addresses as argument to rpc methods
* implement `AddressStrings`
* make a doc clearer
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a finalized state txids query
* Add an address transaction IDs query, without height filters
* Connect the address transaction ID query to the RPC
* Basic filtering of address transaction IDs by height range
* Add a network and range argument to the getaddresstxids test
* Test all block range combinations for mainnet
* Fix a file descriptor limit error
* Optimise seeking the first transaction for an address
The first transaction's location is part of the address location.
* Filter finalized address transaction IDs by height range
* Filter non-finalized address transaction IDs by the height range
* Fix up snapshot tests for the new height range API
* Add `Amount::serialize_as_string` helper method
A helper method that makes it easier to serialize an `Amount` as a
string. This is needed for the response type of the `getaccountbalance`
RPC.
* Implement state service call for address balance
Add `Read{Request,Response}::AddressBalance` variants and implement the
handler that calls the query function.
* Create an `AddressBalance` response type
Only contains the `balance` field which is needed by `lightwalletd`.
That field is serialized as a string, following the RPC specification.
* Implement `get_address_balance` RPC
Query the read-only state service for the information, and wrap it in an
`AddressBalance` response type so that it is serialized correctly.
* Run `rustfmt` inside `proptest!` block
Fix some minor formatting details.
* Test `get_address_balance` with valid addresses
Check that the RPC leads to a query to the mocked state service for a
balance amount.
* Test `get_address_balance` with invalid addresses
An error message should be returned by the RPC.
* Rename metric to `address_balance`
Keep it consistent with how it's named in other places.
Co-authored-by: teor <teor@riseup.net>
* Revert "Add `Amount::serialize_as_string` helper method"
This reverts commit 01b432e3d2.
* Serialize amount as an integer
This is different from what the documentation says, but it's what
lightwalletd expects.
* Add reference to RPC documentation
Make sure it is linked to for easy access.
* Create an `AddressStrings` type
To be used as the input for the `get_address_balance` RPC method.
* Use `AddressStrings` in `get_address_balance` RPC
Fix the input parameter so that the list of address strings is placed
inside a JSON map.
* Update property tests to use `AddressStrings`
Make sure the proper input type is created.
Co-authored-by: teor <teor@riseup.net>
* implement `getaddresstxids` rpc method with dummy empty response
* use already public function
* fix some docs
* pass a list of addresses to the state request
* sync range errors with zcashd
* refactor a loop
* fix grammar
* fix tests
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* Derive Hash for transparent address index types
* Expose some types used by transparent address indexes
* Add an empty transparent transfers type for transparent address indexes
* Update TransparentTransfers with created UTXOs
* Add spent transparent outputs to ContextuallyValidBlock
* Update TransparentTransfers with spent transparent outputs
* Ignore missing spent outputs, so that tests pass
* Remove empty TransparentTransfers after a spend revert
* Update TransparentTransfers with creating and spending transaction IDs
* Ignore duplicate created UTXOs, so that tests pass
* Add some TODO comments
* Remove accidental doctest formatting
* Add address transfers index accessor methods
* Use TransactionLocation in the non-finalized state
* Apply more address index assertions to production code
* Refactor deeply nested code and apply more assertions
* Return UTXOs in chain order
* Return transaction hashes in chain order
* Stop indexing each transparent output multiple times
* Run some more asserts during tests
* Tidy TODO comments
* Fix an incorrect assert condition
* Use OrderedUtxos so that spent UTXOs can be stored in chain order
* Update tests to use OrderedUtxos
* Update the index API for the getaddressutxos query
* Remove redundant arguments in tests
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Change OutputLocation to contain a TransactionLocation
* Change OutputLocation reads from the database
* Update some doc comments
* Update some TODOs
* Change deleting spent UTXOs and updating spent balances
* Change adding new UTXOs and adding their values to balances
* Disable dead code warnings
* Update snapshot test code
* Update round-trip tests for OutputLocations
* Update snapshot test data
* Increment the database format version
* Remove a redundant try_into()
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Refactor redundant code
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* ci: attempt at fixing 'Regenerate stateful disks'
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Split out ReadRequest and ReadResponse state service enums
* Simplify RPC test vectors
* Split state requests into Request and ReadRequest
* Make zebra-rpc use the new state ReadRequest
* Add request metrics to ReadStateService
Add a field to distinguish StateService and ReadStateService metrics.
Add missing StateService request metrics.
* Refactor state transaction lookup so it can be shared between services
* Implement ReadState Request::Transaction
* Revert "Drop the Chain watch channel lock before accessing the finalized state"
This reverts commit 8870944d13.
* Add a WatchReceiver wrapper that always clones the borrowed watch data
This avoids deadlocks, by holding the read lock for as short a time as possible.
* Drop the shared Arc<Chain>s as quickly as possible
This reduces memory usage.
* Make read::block more flexible, by accepting any AsRef<Chain>
* Make the block method docs consistent
* Avoid livelocks by explicitly dropping the borrow after the clone
* Use the read-only state service for RPCs
* Refactor non-finalized block lookup into Chain
* Implement the read-only state block request
* Drop the Chain watch channel lock before accessing the finalized state
* Add state service module docs and cleanup
* Move and add finalized state methods
* Add chain and non-finalized state methods
* Cleanup methods and imports
* Create a ReadStateService type
* Add a stub service implementation
* Add a TODO
* Update ReadStateService request stubs with RPC names and tickets
* Documentation updates
* Make RPC State generic bounds accept a buffered state and a read-only state
* Doc updates
* Add missing proptest-impl feature in RPC dev dependencies
* Move the legacy chain check to the `check` module
And move `populated_state` to the `arbitrary` module.
* Cleanup imports
* Document the state service struct
* Split state block iter into its own module
* feature(rpc): start adding a `getblock` method
* fix(rpc): replace oneshot
* fix(rpc): replace a panic with error
* fix(rpc): fix test
* feature(rpc): add hex to response
* refactor(rpc): use generic instead of alias
* docs(rpc): improve docs for getblock method
* test(rpc): add a test for getblock method
* deps(rpc): remove non needed tower features
Co-authored-by: teor <teor@riseup.net>
* docs(rpc): add a note to getblock doc
* refactor(rpc): replace alias
* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format
* tests(rpc): add network argument to `populated_state()`
* refactor(rpc): use an error for state service readiness
* fix(rpc): add parameter
* fix(rpc): clippy
* nit(rpc): remove new line from imports
* fix(rpc): remove commented code
* fix(rpc): simplify error
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use a `SerializedBlock` type to help serializing blocks (#3725)
* Create a `SerializedBlock` helper type
Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.
* Use `into_iter` instead of `iter`
There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.
* Use `SerializedBlock` in `GetBlock`
Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.
The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.
* feature(rpc): add suggestions from code review
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* tests(rpc): make sure mempool has no requests in get_block test
* fix(rpc): change height argument type in getblock method
* fix(rpc): rustfmt
* fix(rpc): replace panic
* fix(rpc): change getblock response
* fix(rpc): fix lightwalletd test
* tests(rpc): add a getblock error test
* fix(rpc): try another regex
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* feat(log): log the state tip height as part of sync progress logs
* fix(log): downgrade some verbose state logs to debug
* feat(log): log successful gossiped block verification at info level
These logs help us diagnose slow progress near the tip.
There won't be very many of these logs,
because they only happen near the tip.
* fix(log): spawn top-level tasks within the global Zebra tracing span
* fix(log): spawn blocking top-level tasks within the global Zebra tracing span
Co-authored-by: teor <teor@riseup.net>
* lint: enable more clippy checks for bug-prone code
* fix(lint): stop denying lints, to avoid being excluded from Crater
Also categorise lints.
* lint: add some lints to the TODO list
* refactor(arithmetic): partial fixes for some integer arithmetic lints
* Document some weird lint behaviour
* Add Transaction::sprout_joinsplits()
* Add Anchor variants to ValidateContextError
* Make Chain anchor collections pub(crate)
* tracing::instrument several methods in state
* Add contains_*_anchors methods to FinalizedState
* Add check::anchors module and function
* Verify that anchors_refer_to_earlier_treestates in when updating chains in non-finalized state
* Update zebra-state/src/service/check/anchors.rs
Co-authored-by: teor <teor@riseup.net>
* Add anchors() to sapling::ShieldedData
* Add sapling_anchors() to Transaction
* Use Transaction::sapling_anchors() in the anchors_refer_to_earlier_treestates() check
* Whoops, itertools
* Add a comment for improvement
Co-authored-by: teor <teor@riseup.net>
* Add & use a cfg(test) method on FinalizedState to prep test state with anchors to allow other tests to pass contextual checks
* Allow test nullifier checks to pass by populating anchor sets, allowing test anchor checks to pass
* Add mainnet block 419202 and its sapling note commitment tree root to test vectors
* Test sapling anchor verification using the first few Sapling blocks data
* Correct comment
* assert_eq instead of assert(bool)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Update zebra-state/src/service/non_finalized_state.rs
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
There are a lot of these messages when Zebra starts up.
They might be slowing down CI and causing timeouts.
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Check for state errors before updating metrics
Previously, the metrics would be updated for some rejected blocks.
* Clarify and expand block verification metrics
Rename checkpoint-specific metrics to clarify their purpose.
Add metrics for:
- finalized blocks on disk
- blocks verified using the full block verifier
(this metric was previously incorrectly called `zcash_chain_verified_block_height`)
* Update dashboard metric names
Also:
- add some extra block height metrics
- fix a dashboard name
* Add exact block heights to Grafana dashboards
* Add a missing comment
* grafana: use 0 decimals for metrics
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* grafana: show the entire height instead of abbreviated
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* grafana: show the entire height instead of abbreviated
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Fix typo in metric name
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Move height gauges to the state, so they are correct
If we update height gauges in futures, they can execute out of order,
so the metrics can be incorrect.
Instead:
- move the height gauges to the state, and update them based on the best tip
- move the verified block counts to the state
- continue to include all verified blocks on all non-finalized chains
(not just the best chain)
* Show exact checkpoint heights in the dashboard
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
teor
Arya
Marek
Alfredo Garcia
Janito Vaqueiro Ferreira Filho
Conrado Gouvea
Dimitris Apostolou
Deirdre Connolly