* populate `blockcommitmenthash` and `defaultroot` missing fields
* remove assertion line manually from snaps
* fix some imports and docs
* fix some docs
* add a consistency check
* Rename a constant to FINALIZED_STATE_QUERY_RETRIES and use it everywhere
* Move tip query inside retry, split tip into tip_height and tip_hash
* Return retry failures rather than panicking
* Query relevant chain inside the retry
* Check the entire state for consistency, not just the finalized tip
Co-authored-by: teor <teor@riseup.net>
* Mine both standard and min difficulty blocks on testnet
* Add a POW_ADJUSTMENT_BLOCK_SPAN and fix an incorrect assertion
* Split the testnet adjustment into its own function
* Clarify a panic message
* Fix comments
* Re-apply: add acceptance test for getblocktemplate method in CI (#5653)
Revert "Revert "change(tests): add acceptance test for getblocktemplate method in CI (#5653)" (#5672)"
This reverts commit 6446e0ec1b.
* Fix incorrect MAX_CONTEXT_BLOCKS assertion in state
* Actually negate the miner fee for the RPC output
* Try the RPC again after waiting for transactions to verify
* Log before the test waits for the mempool to verify transactions
* Use the new ssh key secrets in CI
* parallelize anchors checks for blocks
* parallelize anchors checks for unmined_tx
* reverts par_iter in block_sapling_orchard_anchors_refer_to_final_treestates
* moves fetch_sprout_final_treestates out of rayon thread
* Remove an unused async track_caller which will soon become a warning
* Explicitly drop unused futures
* Work around a compiler panic (ICE) with flat_map()
https://github.com/rust-lang/rust/issues/105044
* Remove a redundant into_iter()
* allow(clippy::needless_collect)
* updates comments
* adds check nullifier no dup fns for transactions
* Adds:
- check::anchors fn for tx iter
- TODO comments for unifying nullifiers and anchors checks
- new state request
Updates unknown anchor errors to accomodate tx-only check
Calls new state fn from transaction verifier
* updates check::anchors fns to use transactions
updates TransactionContextualValidity request to check sprout anchors
adds comment mentioning TransactionContextualValidity ignores UTXOs
* conditions new state req call on is_mempool
updates tests
* fix doc link / lint error
* checks for duplicate nullifiers with closures
* Update zebra-state/src/service/check/nullifier.rs
Co-authored-by: teor <teor@riseup.net>
* documents find_duplicate_nullifier params
moves if let statement into for loop
* renames new state req/res
* asserts correct response variant in tx verifier
* adds CheckBestChainTipShieldedSpends call in tx verifier to async checks
* re-adds tracing instrumentation to check::anchors fn
renames transaction_in_state to transaction_in_chain
* adds block/tx wrapper fns for anchors checks
* uses UnminedTx instead of transaction.hash()
deletes broken test
* updates new state req/res name
* updates tests and uses par_iter for anchors checks
* Updates check::anchors pub fn docs.
* Adds:
- comments / docs
- a TransactionError variant for ValidateContextError
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* moves downcast to From impl
rustfmt
* moves the ValidateContextError into an Arc
updates comments and naming
* leaves par_iter for another PR
* puts io::Error in an Arc
* updates anchors tests to call tx_anchors check
* updates tests to call tx_no_duplicates_in_chain
slightly improves formatting
* Update zebra-consensus/src/error.rs
Co-authored-by: teor <teor@riseup.net>
* moves Arc from HistoryError to ValidateContextError
Co-authored-by: teor <teor@riseup.net>
* populate block height
* populate cur_time
* populate min_time
* populate capabilities
* populate last_block_hash
* create read state request for getblocktemplate
* refactor to get difficulty fields more properly
* populate bits and target fields
* fix tests
* add target and bits documentation
* docs
* fix docs
* docs
* remove metrixs counter calls
* apply some suggestions from code review
* hide some code behind feature
* simplify the service
* fix error handling
* remove comment
* fox doc
* panic if we dont have enough state
* bring tip data from the state
* make proposal empty
* fix time
* fix docs, consensus rules
* remove non used anymore fn
* remove another non used fn
* remove no needed change
* remove more unused changes
* remove unused anymore change
* apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* fix build and snapshots
* apply testnet consensus rule
* fix test
* rustfmt
* remove time as allowed field to be modified by the miner if mining minimum difficulty block
* move all times to before calculating difficulty
* do some cleanup
* Adjust times so the whole time range is a testnet minimum difficulty block
* Return a GetBlockTemplateChainInfo struct from the difficulty calculation
* Add a Zebra-only max_time field to the getblocktemplate RPC
Co-authored-by: teor <teor@riseup.net>
* Uses BestChainUtxo to find utxos for mempool
* adds missing input test
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* update other instances of the renamed InputNotFound error
* adds read::unspent_utxo fn
* adds test for success case
Co-authored-by: teor <teor@riseup.net>
* Use correct release for getblocktemplate config
* Include at least 2 full checkpoints in the lookahead limit
* Increase full sync timeout to 36 hours
* Only log "synced block height too far ahead of the tip" once
* Replace AboveLookaheadHeightLimit error with pausing the syncer
* Use AboveLookaheadHeightLimit for blocks a very long way from the tip
* Also add the getblocktemplate config, and fix the test message
* Remove an outdated TODO comment
* Allow syncing again when a small number of blocks are in the queue
* Allow some dead code
* Fix error text for state service for syncer
* Fix error handling in syncer
* cargo fmt --all
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add documentation for the getblocktemplate RPC
* Add a new mempool::Request::Transactions
* Add conversions from Vec<UnminedTx> to merkle::Root and AuthDataRoot
* Fill in the merkle root and auth data root fields
* Delete the Coinbase type, it's the same as Transaction
* Fill in some other existing types
* Add Hex serialization support to some zebra-chain types
* Add TransactionTemplate fields and fill some in
* Fix test hangs by spawning async tasks
* Add temporary workaround for no transactions in the block
* Encode hashes and roots as hex
* Update RPC snapshots
* Add a missing Request::Transactions handler
* Fix doc warnings
* Fix fee serialization
* Update snapshots for serialization changes
* Add a missing Cargo.lock change
* Change depends type
* Remove duplicate feature entry
* Document the new RPC feature
* Fix a comment typo
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Update default roots docs
* Fix comment typo
* Fix a comment typo
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Arya <aryasolhi@gmail.com>
* adds transactions to ChainTipBlock and rejects transactions from the mempool that have been invalidated by the latest block commit
* merges remove_same_effects in with reject_invalidated_transactions, moves nullifier retrieval to the new Storage method, rejects mined_ids, and updates tests
* updates DuplicateSpend's error message
* fixes tests
* Update zebrad/src/components/mempool/storage.rs
Co-authored-by: teor <teor@riseup.net>
* adds comment
* formatting for the proptest
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* Add minimum queued block height to metrics
* Add metrics to sent block cache
* Add metrics for the last sent finalized height
* Refactor sent metrics
* Add new block channel metrics to a grafana dashboard
* Add a new block commit task and channels, that don't do anything yet
* Add last_block_hash_sent to the state service, to avoid database accesses
* Update last_block_hash_sent regardless of commit errors
* Rename a field to StateService.max_queued_finalized_height
* Commit finalized blocks to the state in a separate task
* Check for panics in the block write task
* Wait for the block commit task in tests, and check for errors
* Always run a proptest that sleeps once
* Add extra debugging to state shutdowns
* Work around a RocksDB shutdown bug
* Close the finalized block channel when we're finished with it
* Only reset state queue once per error
* Update some TODOs
* Add a module doc comment
* Drop channels and check for closed channels in the block commit task
* Close state channels and tasks on drop
* Remove some duplicate fields across StateService and ReadStateService
* Try tweaking the shutdown steps
* Update and clarify some comments
* Clarify another comment
* Don't try to cancel RocksDB background work on drop
* Fix up some comments
* Remove some duplicate code
* Remove redundant workarounds for shutdown issues
* Remode a redundant channel close in the block commit task
* Remove a mistaken `!force` shutdown condition
* Remove duplicate force-shutdown code and explain it better
* Improve RPC error logging
* Wait for chain tip updates in the RPC tests
* Wait 2 seconds for chain tip updates before skipping them
* Remove an unnecessary block_in_place()
* Fix some test error messages that were changed by earlier fixes
* Expand some comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
* Actually drop children of failed blocks
* Explain why we drop descendants of failed blocks
* Clarify a comment
* Wait for chain tip updates in a failing test on macOS
* Clean duplicate finalized blocks when the non-finalized state activates
* Send an error when receiving a duplicate finalized block
* Update checkpoint block behaviour, document its consensus rule
* Wait for chain tip changes in inbound_block_height_lookahead_limit test
* Wait for the genesis block to commit in the fake peer set mempool tests
* Disable unreliable mempool verification check in the send transaction test
* Appease rustfmt
* Use clear_finalized_block_queue() everywhere that blocks are dropped
* Document how Finalized and NonFinalized clones are different
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* updates comments, renames send_process_queued, other minor cleanup
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* updates comments, renames send_process_queued, other minor cleanup
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* removes duplicate field definitions on StateService that were a result of a bad merge
* update NotReadyToBeCommitted error message
* Appear rustfmt
* Fix doc links
* Rename a function to initial_contextual_validity()
* Do error tasks on Err, and success tasks on Ok
* Simplify parent_error_map truncation
* Rewrite best_tip() to use tip()
* Rename latest_mem() to latest_non_finalized_state()
```sh
fastmod latest_mem latest_non_finalized_state zebra*
cargo fmt --all
```
* Simplify latest_non_finalized_state() using a new WatchReceiver API
* Expand some error messages
* Send the result after updating the channels, and document why
* wait for chain_tip_update before cancelling download in mempool_cancel_mined
* adds `sent_non_finalized_block_hashes` field to StateService
* adds batched sent_hash insertions and checks sent hashes in queue_and_commit_non_finalized before adding a block to the queue
* check that the `curr_buf` in SentHashes is not empty before pushing it to the `sent_bufs`
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix rustfmt
* Check for finalized block heights using zs_contains()
* adds known_utxos field to SentHashes
* updates comment on SentHashes.add method
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* return early when there's a duplicate hash in QueuedBlocks.queue instead of panicking
* Make finalized UTXOs near the final checkpoint available for full block verification
* Replace a checkpoint height literal with the actual config
* Update mainnet and testnet checkpoints - 7 October 2022
* Fix some state service init arguments
* Allow more lookahead in the downloader, but less lookahead in the syncer
* Add the latest config to the tests, and fix the latest config check
* Increase the number of finalized blocks checked for non-finalized block UTXO spends
* fix(log): reduce verbose logs for block commits (#5348)
* Remove some verbose block write channel logs
* Only warn about tracing endpoint if the address is actually set
* Use CloneError instead of formatting a non-cloneable error
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Increase block verify timeout
* Work around a known block timeout bug by using a shorter timeout
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a new block commit task and channels, that don't do anything yet
* Add last_block_hash_sent to the state service, to avoid database accesses
* Update last_block_hash_sent regardless of commit errors
* Rename a field to StateService.max_queued_finalized_height
* Commit finalized blocks to the state in a separate task
* Check for panics in the block write task
* Wait for the block commit task in tests, and check for errors
* Always run a proptest that sleeps once
* Add extra debugging to state shutdowns
* Work around a RocksDB shutdown bug
* Close the finalized block channel when we're finished with it
* Only reset state queue once per error
* Update some TODOs
* Add a module doc comment
* Drop channels and check for closed channels in the block commit task
* Close state channels and tasks on drop
* Remove some duplicate fields across StateService and ReadStateService
* Try tweaking the shutdown steps
* Update and clarify some comments
* Clarify another comment
* Don't try to cancel RocksDB background work on drop
* Fix up some comments
* Remove some duplicate code
* Remove redundant workarounds for shutdown issues
* Remode a redundant channel close in the block commit task
* Remove a mistaken `!force` shutdown condition
* Remove duplicate force-shutdown code and explain it better
* Improve RPC error logging
* Wait for chain tip updates in the RPC tests
* Wait 2 seconds for chain tip updates before skipping them
* Remove an unnecessary block_in_place()
* Fix some test error messages that were changed by earlier fixes
* Expand some comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
* Actually drop children of failed blocks
* Explain why we drop descendants of failed blocks
* Clarify a comment
* Wait for chain tip updates in a failing test on macOS
* Clean duplicate finalized blocks when the non-finalized state activates
* Send an error when receiving a duplicate finalized block
* Update checkpoint block behaviour, document its consensus rule
* Wait for chain tip changes in inbound_block_height_lookahead_limit test
* Wait for the genesis block to commit in the fake peer set mempool tests
* Disable unreliable mempool verification check in the send transaction test
* Appease rustfmt
* Use clear_finalized_block_queue() everywhere that blocks are dropped
* Document how Finalized and NonFinalized clones are different
* Use the same check as commit_finalized() for finalized block heights
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* use spawn_blocking to run zebra_state::init from start cmd
* uses zebra_state::spawn_init in copy-state command
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Look back 10,000 blocks on testnet for a legacy chain
* Use a smaller number of legacy chain blocks in the tests
* Fix test assertion error message
* Clarify legacy chain check comment
* cargo fmt --all
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move AwaitUtxos next to the other shared writeable state requests
* Rename ReadResponse::Utxos to ReadResponse::AddressUtxos
```sh
fastmod Utxos AddressUtxos zebra*
```
* Rename an out_point variable to outpoint for consistency
* Rename transparent_utxos to address_utxos
```sh
fastmod transparent_utxos address_utxos zebra*
```
* Run AwaitUtxo without accessing shared mutable chain state
* Fix some incorrect comments
* Explain why some concurrent reads are ok
* Add a TODO
* Stop using self.mem in AwaitUtxo requests
* Update state service module documentation
* Move the QueuedBlock type into the queued_blocks module
* Explain how spent UTXOs are treated by the state
* Clarify how cached Chains impact state read requests
And move repeated comments to the module header.
* fastmod ChainUtxo BestChainUtxo zebra*
* Add an AnyChainUtxo request
* Make AwaitUtxo non-blocking
* Move the finalized block queue into the StateService
* Move the queued_blocks module to the state service
* Move QueuedFinalized into queued_blocks
* Move the queued_blocks tests into their own module
* Make the FinalizedState cloneable
* cleanup of repetitive code
* fixes merge by adding back req.count_metric and removing the metrics::counter in AwaitUtxo
Co-authored-by: teor <teor@riseup.net>
* Move the finalized block queue into the StateService
* Move the queued_blocks module to the state service
* Move QueuedFinalized into queued_blocks
* Move the queued_blocks tests into their own module
* Make the FinalizedState cloneable
* Move AwaitUtxos next to the other shared writeable state requests
* Rename ReadResponse::Utxos to ReadResponse::AddressUtxos
```sh
fastmod Utxos AddressUtxos zebra*
```
* Rename an out_point variable to outpoint for consistency
* Rename transparent_utxos to address_utxos
```sh
fastmod transparent_utxos address_utxos zebra*
```
* Run AwaitUtxo without accessing shared mutable chain state
* Fix some incorrect comments
* Explain why some concurrent reads are ok
* Add a TODO
* Stop using self.mem in AwaitUtxo requests
* Update state service module documentation
* Move the QueuedBlock type into the queued_blocks module
* Explain how spent UTXOs are treated by the state
* Clarify how cached Chains impact state read requests
And move repeated comments to the module header.
* fastmod ChainUtxo BestChainUtxo zebra*
* Add an AnyChainUtxo request
* Make AwaitUtxo non-blocking
* Add TODOs for finalized blocks, non-finalized blocks, and reads
* Document how state requests use shared state
* Add a ReadStateService to the StateService
And cleanup service struct fields.
* Redirect Block and Transaction Requests to the ReadStateService
* Put AddressBalance in a consistent enum order
* Turn a repeated comment into documentation
* Tidy doc links
* Run Tip requests concurrently
* Remove some redundant timers
* Run Depth requests concurrently
* Run BlockLocator requests concurrently
* Move BlockLocator tests
* Run FindBlockHashes requests concurrently
* Run FindBlockHeaders requests concurrently
* Use a constant in documentation
Co-authored-by: Marek <mail@marek.onl>
* Link that constant correctly
* Expand block_locator() documentation
* Clarify the difference between tower::Buffers and the state's ordered queues
* Explain block locators better
Co-authored-by: Marek <mail@marek.onl>
* adds non-blocking writer for tracing subscriber
* use non_blocking writer for the fmt::Layer with the tokio-console feature as well
* adds doc comment to _guard field
* adds acceptance test
* update filter_handle type to use NonBlocking
* adds more detail on lossy non-blocking writer and sets tracing.filter to "trace" in acceptance test
* drops ZebradApp before process::exit(1) in the event of a FrameworkError
* reduces buffered lines limit to 8000
* adds tracing.buffer_limit config and some comments
* update acceptance.rs
* fix acceptance test
* fixes ambigious phrasing in comment
* updates zebrad/src/application.rs
* Find out what the join error is in the GitHub runner tests
* updates acceptance test to use recv_timeout instead of always waiting 10 seconds, removes unnecessary echo command, and reduces # of rpc requests to 500
* see if sleeping for a few seconds before exiting helps the macOS test pass
* Expand exit sleep docs
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Add history trees for each height in non-fin state
* Refactor formatting
* Pass the treestate to the finalized state
I created a new structure `FinalizedBlockWithTrees` that wraps the
treestate and the finalized block. I did that because the original
`FinalizedBlock` is `Eq`, but `HistoryTree` can't be `Eq`.
This makes Zebra faster because:
1. The finalized state doesn't retrieve the treestate from the disk if
the non-finalized state supplies it.
2.The finalized state doesn't recompute the treestate if the
non-finalized state supplies it.
* Check block commitment before updating hist tree
* Store Sprout commitment trees in non-fin state
* Send trees for the root block to fin-state
When committing a block and sending the treestate from the non-finalized
state to the finalized state, Zebra was sending trees that correspond to
the tip block instead of trees that correspond to the root block of the
best chain. This commit fixes that.
* Refactor doc comments
* Refactor block finalization
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* cargo upgrade --workspace rocksdb
* Add a deny.toml exception for bindgen 0.59
* Move `valid()` and `status()` methods to raw iterators
* Update some outdated comments
* Panic on iterator failures
We might want to change this if there are any common failure modes.
* allow(clippy::unwrap_in_result) in some methods
* Rename a function to prepare_block_header_and_transaction_data_batch()
* Fix formatting of test command timeouts and child process output
* Put some #[cfg()]s in the standard Rust location
* Update some test timings
* Allow code timers to be ignored
* Revert: deserialize stored transactions in a rayon thread
* Add a TODO for the reverted bug fix
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add history trees for each height in non-fin state
* Refactor formatting
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Fix clippy::let_and_return
* Increase lightwalletd test timeouts for zebrad slowness
* Add a `zebrad_update_sync()` test, that update syncs Zebra without lightwalletd
* Run the zebrad-update-sync test in CI
* Add extra zebrad time to workaround lightwalletd bugs
* Add a CodeTimer diagnostic struct for long-running code
* Time state init and each state request, log when it takes too long
* Add code locations to execution timers
* Instrument state futures and functions with tracing spans
* Only log each code timer once
* Make displayed times shorter
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography
* Improve anchor validation debugging and error messages
* Work around a test data bug, and save some CPU
* Remove redundant checks for empty shielded data
* Skip generating unused interstitial treestates
* Do disk fetches and quick checks, then CPU-heavy cryptography
* Wrap HistoryTree in an Arc in the state
* Run CPU-intensive chain validation and updates in parallel rayon threads
* Refactor to prepare for parallel tree root calculations
* Run finalized state note commitment tree root updates in parallel rayon threads
* Update finalized state note commitment trees using parallel rayon threads
* Fix a comment typo and add a TODO
* Split sprout treestate fetch into its own function
* Move parallel note commitment trees to zebra-chain
* Re-calculate the tree roots in the same parallel batches
* Do non-finalized note commitment tree updates in parallel threads
* Update comments about note commitment tree rebuilds
* Do post-fork tree updates in parallel threads
* Add a TODO for parallel tree updates in tests
* Fix broken intra-doc links
* Clarify documentation for sprout treestates
* Spawn large database reads into blocking tokio threads
* Concurrently read all blocks, headers, and transactions from disk
* Run zebra-state transaction deserialization on a rayon thread
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography
* Improve anchor validation debugging and error messages
* Work around a test data bug, and save some CPU
* Remove redundant checks for empty shielded data
* Skip generating unused interstitial treestates
* Do disk fetches and quick checks, then CPU-heavy cryptography
* Wrap HistoryTree in an Arc in the state
* Run CPU-intensive chain validation and updates in parallel rayon threads
* Refactor to prepare for parallel tree root calculations
* Run finalized state note commitment tree root updates in parallel rayon threads
* Update finalized state note commitment trees using parallel rayon threads
* Fix a comment typo and add a TODO
* Split sprout treestate fetch into its own function
* Move parallel note commitment trees to zebra-chain
* Re-calculate the tree roots in the same parallel batches
* Do non-finalized note commitment tree updates in parallel threads
* Update comments about note commitment tree rebuilds
* Do post-fork tree updates in parallel threads
* Add a TODO for parallel tree updates in tests
* Fix broken intra-doc links
* Clarify documentation for sprout treestates
* Sort Cargo.toml dependencies
* Update comments about note commitment tree rebuilds
* Add info-level logs to time note commitment tree rebuilds
* Log the number of rebuilt blocks after a fork, and the time per block
* Move humantime formats to zebra-chain
* Use human-friendly time formatting
* Only fetch block headers from the database to answer headers requests
* Move writing to the database to a blocking thread
* Add blocking threads to tests that need them
* Remove mempool downloader requirement for a populated state
And stop populating states that don't need it,
so we can use tokio::time::pause() in those tests.
* Improve debugging for an intermittent test failure
* Try to avoid a race condition populating the mempool in tests
* Wrap Sprout note commitment trees into `Arc`
* Remove a redundant comment
* Rephrase a comment about chain forking
* Remove a redundant comment
The comment is not valid because Zebra uses `bridgetree::Frontier`s from
the `incrementalmerkletree` crate to represent its note commitment
trees. This `struct` does not support popping elements from the tree.
* Wrap Sapling commitment trees into `Arc`
* Remove unnecessary `as_ref`s
* Wrap Orchard commitment trees into `Arc`
* Rename `block_by_height` to `block_header_by_height` in fin state
* Rename `tx_by_hash` to `tx_loc_by_hash` in both (non & fin) states
* Rename `utxo_by_outpoint` to `utxo_by_out_loc` in finalized state
* Reorder the column families so that they match the docs
* Update `struct Chain` in the RFCs
* Increment `DATABASE_FORMAT_VERSION` to 25
* Remove obsolete docs from `0004-asynchronous-script-verification.md`
* Remove an obsolete `TODO` from `disk_db.rs`
* Delete unused snapshots
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* delete old database directories
* check if state directory exists
* skip deleting when ephemeral
* split `check_and_delete_old_databases`
* move `check_and_delete_old_databases` to state
* spawn `check_and_delete_old_databases`
* simplity a bit
* fix(state): only delete old database directories inside the cache directory (#4631)
* Add function comments, tweak log
* Simplify version parsing
* Use spawn_blocking to launch the task on a separate thread, do the cleanup last
* Abort the cleanup task when Zebra exits
* Split directory deletion into its own function, handle ownership
* Rename cache_dir to state_dir
* If an outdated state directory is outside the cache directory, don't delete it
* Minimise diffs
* add test
* fix typos
Co-authored-by: teor <teor@riseup.net>
* add `canonicalize` to test regex
* add another match to test
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
teor
Alfredo Garcia
Arya
Marek
Conrado Gouvea