* Adds logging of column family size and database size on startup and shutdown
* Changes log level of column families size strings to debug. Adds TODO comment to use human_bytes crate for human-readable format of metrics. Adds print_db_metrics function to ZebraDb struct.
* Calls enumerate() on column_families to access index var
* Resolves cargo fmt checker results
* Resolves clippy lint
* Runs and fixes changes from fmt, clippy, check and test
* Removes prop.txt
* minor doc changes
---------
Co-authored-by: Elijah Hampton <elijahhampton@MBP-de-Elijah-2.lan>
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* removed derive copy from network, address and ledgerstate
* changed is_max_block_time_enforced to accept ref
* changed NetworkUpgrade::Current to accept ref
* changed NetworkUpgrade::Next to accept ref
* changed NetworkUpgrade::IsActivationHeight to accept ref
* changed NetworkUpgrade::TargetSpacingForHeight to accept ref
* changed NetworkUpgrade::TargetSpacings to accept ref
* changed NetworkUpgrade::MinimumDifficultySpacing_forHeight to accept ref
* changed NetworkUpgrade::IsTestnetMinDifficultyBlock to accept ref
* changed NetworkUpgrade::AveragingWindowTimespanForHeight to accept ref
* changed NetworkUpgrade::ActivationHeight to accept ref
* changed sapling_activation_height to accept ref
* fixed lifetime for target_spacings
* fixed sapling_activation_height
* changed transaction_to_fake_v5 and fake_v5_transactions_for_network to accept ref to network
* changed Input::vec_strategy to accept ref to network
* changed functions in zebra-chain/src/primitives/zcash_history.rs to accept ref to network
* changed functions in zebra-chain/src/history_tree.rs to accept ref to network
* changed functions in zebra-chain/src/history_tree.rs to accept ref to network
* changed functions in zebra-chain/src/primitives/address.rs to accept ref to network
* changed functions in zebra-chain/src/primitives/viewing_key* to accept ref to network
* changed functions in zebra-chain/src/transparent/address.rs to accept ref to network
* changed functions in zebra-chain/src/primitives/zcash_primitives.rs to accept ref to network
* changed functions in zebra-chain/src/primitives/zcash_note_encryption.rs to accept ref to network
* changed functions in zebra-chain/src/primitives/history_tree* to accept ref to network
* changed functions in zebra-chain/src/block* to accept ref to network
* fixed errors in zebra-chain::parameters::network
* fixed errors in zebra-chain::parameters::network
* fixed errors in zebra-chain
* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref
* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref
* fixed errors in zebra-chain/src/block/arbitrary.rs
* finished fixing errors in zebra-chain - all crate tests pass
* changed functions in zebra-state::service::finalized_state to accept &Network
* changed functions in zebra-state::service::non_finalized_state to accept &Network
* zebra-state tests run but fail with overflow error
* zebra-state tests all pass
* converted zebra-network -- all crate tests pass
* applied all requested changes from review
* converted zebra-consensus -- all crate tests pass
* converted zebra-scan -- all crate tests pass
* converted zebra-rpc -- all crate tests pass
* converted zebra-grpc -- all crate tests pass
* converted zebrad -- all crate tests pass
* applied all requested changes from review
* fixed all clippy errors
* fixed build error in zebrad/src/components/mempool/crawler.rs
* begin refactor suggested as "step 2": https://github.com/ZcashFoundation/zebra/issues/7968#issue-2003245309
Squashed from multiple commits to enable partial rebase
* break out more little traits
* add activation implementation leveraging From<Network> for lrz::cons::
* for transfer of ownership I cannot return a type that's owned by the method
* hrp_sapling_extended_full_viewing_key
* complete implementation of interface of Parameters on Network reuse Parameters on zcash Network where possible
* move doc-comments to trait declarations (from impls)
* Simplify/complete Parameters impl for Network
* Add checkpoint_list method, move documentation, etc
* move last match network to inside network method
* add back comment
* use zcash_address for parameter types in zebra-chain
* use inherent methods instead of big parameters passthrough
* revert to implementation of From on zcash_primitives::..::Network vs &zcash_prim...
* move match
* add test to block maximum time rule
* update changelog
* finish porting target_difficutly_limit
* remove obscelete code comment
* revert non-functional change
* finish migrating target_difficulty_limit, checkpoint_list
* update changelog
---------
Co-authored-by: Hazel OHearn <gygaxis@zingolabs.org>
* added functions for fetching block vectors
* inserted new network methods for vector fetching into zebra-chain
* changed tag back to test/feature=branch
* changed tag back to test/feature=branch
* changed tag back to test/feature=branch
* changed feature tag to proptest-impl, started implementing in zebra-consensus tests
* adding methods to zebra-consensus, lifetime error in src/transaction/tests.rs, needs refactoring
* finished adding methods to zebra-consensus
* finished adding new methods to zebrad
* added new methods to zebra-rpc and zebra-scan
* finished removing statements matching on Network from tests
* updated new error message
* changed get_block_bytes() and get_block_sapling_roots_bytes to return option and removed serialization error types as per requested changes in PR review
* removed match statements from zebra_chain::transaction::arbitrary::test_transactions() and new zebra-grpc tests
* moved zebra-chain::test_utils to zebra-chain::test
* removed get_ prefix from getter methods
* renamed zebra-chain::test to zebra-chain::tests
* renamed zebra-chain::test to zebra-chain::tests
* fixed clippy warnings
* changed block_map to return clone
* chore(dev-deps): always specify the version for dev-dependencies
* test(tower-batch-control): remove zebra-consensus dev dependency
Copies the Ed25519Verifier code directly to tower-batch-control tests
* test(zebra-consensus): update ed25519 verifier tests with ones from batch-control
* test(zebra-consensus): restore previous timeout values
The timeouts copied from tower-batch-control seems too low
and there must have been a reasoning why were they introduced
* chore: update dev-deps versions to beta.33
* chore(tower-batch-control): remove dev-dependency on metrics
* chore(tower-batch-control): remove zebra-chain dev-dependency
* Update zebra-scan/Cargo.toml
* bump all versions to match current release
* fix missed commas in version bumps
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
* Patch equihash to use the solver branch
* Add an internal-miner feature and set up its dependencies
* Remove 'Experimental' from mining RPC docs
* Fix a nightly clippy::question_mark lint
* Move a byte array utility function to zebra-chain
* fixup! Add an internal-miner feature and set up its dependencies
* Add an equihash::Solution::solve() method with difficulty checks
* Check solution is valid before returning it
* Add a TODO to check for peers before mining
* Move config validation into GetBlockTemplateRpcImpl::new()
* fixup! fixup! Add an internal-miner feature and set up its dependencies
* Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls
* Start adding an internal miner component
* Add the miner task to the start command
* Add basic miner code
* Split out a method to mine one block
* Spawn to a blocking thread
* Wait until a valid template is available
* Handle shutdown
* Run mining on low priority threads
* Ignore some invalid solutions
* Use a difference nonce for each solver thread
* Update TODOs
* Change the patch into a renamed dependency to simplify crate releases
* Clean up instrumentation and TODOs
* Make RPC instances cloneable and clean up generics
* Make LongPollId Copy so it's easier to use
* Add API to restart mining if there's a new block template
* Actually restart mining if there's a new block template
* Tidy instrumentation
* fixup! Move config validation into GetBlockTemplateRpcImpl::new()
* fixup! Make RPC instances cloneable and clean up generics
* Run the template generator and one miner concurrently
* Reduce logging
* Fix a bug in getblocktemplate RPC tip change detection
* Work around some watch channel change bugs
* Rate-limit template changes in the receiver
* Run one mining solver per available core
* Use updated C code with double-free protection
* Update to the latest solver branch
* Return and submit all valid solutions
* Document what INPUT_LENGTH means
* Fix watch channel change detection
* Don't return early when a mining task fails
* Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness
* Make existing parallelism docs and configs consistent
* Add a mining parallelism config
* Use the minimum of the configured or available threads for mining
* Ignore optional feature fields in tests
* Downgrade some frequent logs to debug
* Document new zebrad features and tasks
* Describe the internal-miner feature in the CHANGELOG
* Update dependency to de-duplicate equihash solutions
* Use futures::StreamExt instead of TryStreamExt
* Fix a panic message typo
* impl TryFrom<zcash_primitives::BlockHeight> for Height
* Add type-safe read and write database methods
* Only allow typed access to the scanner DB
* Update docs
* Implement a common method as a trait
* Fix imports
* Tidy state imports
* Activate tracing logging macros in the whole scanner crate
* Fix dead code warnings
* Use a more sensible export order
* Remove a 1.72 lint exception now 1.74 is stable
* Switch history trees over to TypedColumnFamily, and remove redundant code
* Add typed batch creation methods, and switch history trees to them
* Convert ValueBalance to typed column families
* Make the APIs compatible after a merge
* Use `ZebraDb` instead of `DiskDb` where needed
---------
Co-authored-by: Marek <mail@marek.onl>
* impl TryFrom<zcash_primitives::BlockHeight> for Height
* Add type-safe read and write database methods
* Only allow typed access to the scanner DB
* Update docs
* Implement a common method as a trait
* Fix imports
* Tidy state imports
* Activate tracing logging macros in the whole scanner crate
* Fix dead code warnings
* start scanner where it was left
* fix tests
* add a `scan_start_where_left` test
* refactor a log msg
* fix some comments
* remove function
* fix doc comment
* clippy
* fix `sapling_keys_and_last_scanned_heights()`
* simplify start height
* i went too far, revert some changes back
* change log info to every 10k blocks
* fix build
* Update height snapshot code and check last height is consistent
* Add strictly before and strictly after database key gets
* Move to the previous key using strictly before ops
* Assert that keys are only inserted once
* Update the index in each loop
* Update snapshots
* Remove debugging code
* start scanning at min available height
---------
Co-authored-by: teor <teor@riseup.net>
* Refactor test data into functions
* Add a typed snapshot test for scanner storage
* Use standard hex serialization with SaplingScannedResult
* Simplify transaction::Hash hex serialization
* Sort HashMaps before snapshotting
* Add typed snapshot data
* Make some scanner storage methods more flexible
* Move tests to a submodule and expose test functions and constants
* Make scanner functions clearer and easier to use
* Simplify state snapshot test code
* Add raw data snapshot tests for the scanner
* Add snapshots
* Fix import path
* Fix import conditional compilation
* fix imports
* fix imports 2
* Put read and write db exports together
* Remove confusing IntoDisk/FromDisk impl
* Fix an incorrect unused method that could panic
* Delete a test that is no longer valid
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Upgrade the scanner database major version to 1
* Update format docs
* Change the high-level scanner db format
* Change the scanner serialization formats
* Fix value format and tests
* Fix incorrect types
* Update documentation
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Fix availability of tokio::time in scanner
* Create a function that parses a key into a list of keys
* Pass a ChainTipChange to the scanner function
* Convert a scanned block to a sapling result
* Make it easier to pass keys and blocks
* Increase scanner wait times
* Parse keys once at the start of the scan
* Get a block from the state instead of the tip
* Don't log secret keys, only log every 100,000 blocks
* Scan each block and add the results to storage
* Move blocking tasks into spawn_blocking()
* Update the acceptance test
* Use a dummy sapling tree size
* Use a larger dummy size
* Expose IntoDisk and FromDisk in zebra-state
* Implement database serialization for SaplingScanningKey Strings
* Implement serialization for Vec<SaplingScannedResult> (Vec<transaction::Hash>)
* Implement seralization for SaplingScannedDatabaseIndex
* Add an is_empty() method
* Add a read method for a specific index, and document it
* Implement writing scanner results to the database
* Make read name more explicit
* Implement writing scanner keys
* Implement reading sapling keys
* Spawn blocking tasks correctly in async code
* Change storage results methods to use the database
* Update tests that use storage
* Use spawn_blocking() for database methods
* Change the check interval to slightly less than the block interval
* Expose raw database methods with shielded-scan
* fix `scan_task_starts` test
* minor doc change in test
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Create an empty storage/db module
* Use ephemeral storage in tests
* Populate storage inside new() method
* Move scanner setup into an init() method
* Pass the network to scanner init
* Create a database but don't actually use it
* Skip shutdown format checks when skipping format upgrades
* Allow the scanner to skip launching format upgrades in production
* Refactor skipping format upgrades so it is consistent
* Allow checking configs for equality
* Restore Network import
* Provide access to DiskDb and DiskWriteBatch outside the state using a feature
* Actually let's export ZebraDb for the format upgrade code
* Pass column families to ZebraDb as an argument
* Allow the database kind to be changed in config.rs
* Use the state kind in finalized_state.rs
* Allow different database kinds in ZebraDb, but don't move the upgrade code yet
* Allow different database kinds in DiskDb
* Allow different database kinds in upgrade.rs, but don't split the upgrade code out yet
* Add new arguments to raw database tests
* Fix doc links
* Fix internal imports
* Fix unused code
* Update zebrad version metadata
* Create a specific state database delete function
* Fix state exports
* Fix zebrad tests
* Fix zebrad state write tests
* Make CI run again
* Fix dead code warnings for test methods
* Remove unnecessary async on some tests
* Fix logging required by tests
* Fix logging required in test itself
* Fix variable names
* Try to copy the message and add regexes
* Avoid a race condition in the 25.3.0 state upgrade
* Look for the most recent sprout tree height if needed
* Get the latest history tree not the tip height history tree
* Discard keys provided by low level database method
* Remove extra ;
* Provide key types & rustfmt
* Fix weird closure type syntax
* Update zebra-state/src/service/finalized_state/disk_format/upgrade/fix_tree_key_type.rs
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* Refactor return type of poll_discover()
* Simplify poll_ready() by removing preselected peers
* Fix peer set readiness check
* Pass task context correctly to background tasks
* Make poll_discover() return Pending
* Make poll_inventory() return Pending
* Make poll_unready() return Poll::Pending
* Simplify with futures::ready!() and ?
* When there are no peers, wake on newly ready peers, or new peers, in that order
* Preserve the original waker when there are no unready peers
* Fix polling docs and remove unnecessary code
* Make sure we're ignoring Poll::Pending not Result
* Make panic checking method names clearer
* Fix connection Client task wakeups and error handling
* Cleanup connection panic handling and add wakeup docs
* Fix connection client task wakeups to prevent hangs
* Simplify error and pending handling
* Clarify inventory set behaviour
* Define peer set poll_* methods so they return Ok if they do something
* Clarify documentation
Co-authored-by: Arya <aryasolhi@gmail.com>
* Fix test that depended on preselected peers
* Check ready peers for errors before sending requests to them
* Fix a hanging test by not waiting for irrelevant actions
* Only remove cancel handles when they are required
* fix incorrect panic on termination setting
* Clarify method comments
Co-authored-by: Arya <aryasolhi@gmail.com>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Stop using `SemanticallyVerifiedBlockWithTrees`
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Doc that default treestate corresponds to genesis
* Doc the diff between default values of tree roots
---------
Co-authored-by: teor <teor@riseup.net>
* Simple replacements of doc.zebra.zfnd.org with docs.rs
* Manual fixes for specific main/internal/external docs
* Point developer docs to doc-internal.zebra.zfnd.org
* fastmod --glob '\!.git' -- doc.zebra.zfnd.org/zebrad docs.rs/zebrad/latest/zebrad
* Manually remove any remaining doc.zfnd.zebra.org links
* Remove the external docs job
* Add changelog entry and fix links
* Fix links that were broken before this PR
* refactors sapling_subtrees to use ranges
* refactor orchard_subtrees() to use ranges
* Restores check for inconsistent subtree roots
* updates vectors tests
* Applies some suggestions from code review and adds `read::tree::subtrees` function
* updates correctness comment & db read method names
* adds test_subtrees
* tests in-memory reads
* test that subtree read fns work right with excluded start bound
* applies suggestions from code review
* Applies suggestions from code review.
* Updates docs applying suggestion from code review
* adds new arg to zs_range_iter calls
* Add tree key format and cached root upgrades
* Document the changes in the upgrades
* Remove unnecessary clippy::unwrap_in_result
* Fix database type
* Bump state version
* Skip some checks if the database is empty
* Fix tests for a short state upgrade
* Disable format checks in some tests
* Document state performance issues
* Clarify upgrade behaviour
* Clarify panic messages
* Delete incorrect genesis trees write code
* Fix metrics handling for genesis
* Remove an unused import
* Explain why genesis anchors are ok
* Update snapshots
* Debug a failing test
* Fix some tests
* Fix missing imports
* Move the state check in a test
* Fix comment and doc typos
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
* Clarify what a long upgrade is
* Rename unused function arguments
Co-authored-by: Marek <mail@marek.onl>
* Add all_unordered log regex matching methods
* Fix timing issues with version upgrades and other logs
* Fix argument name in docs
Co-authored-by: Marek <mail@marek.onl>
* Explain match until first for all regexes behaviour better
---------
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
* Uses range_iter in address_transaction_locations
* Uses range_iter in address_transaction_locations
* uses u16::MAX instead of usize::MAX
* Moves limit code into method
* adds allow(dead_code)
* Simplifies address_iterator_range
* Moves test state init out of loop
* Updates docs
* Fix the combining of Sapling subtrees
This commit fixes a bug due to which the function `sapling_subtrees`
used to always include all subtrees from the non-finalized state without
respecting the given limit.
* Fix the combining of Orchard subtrees
This commit fixes a bug due to which the function `orchard_subtrees`
used to always include all subtrees from the non-finalized state without
respecting the given limit.
* Add additional checks when retrieving subtrees
* Allow raw subtree insertions into `Chain` in tests
* Derive `Default` for `Chain`
* Derive `Default` for note commitment tree nodes
* Use `pub(super)` for `DiskWriteBatch`
Set the visibility of `DiskWriteBatch` to `pub(super)`.
* Add tests for retrieving subtrees
* Use `default()` for subtree roots in tests
This change is unrelated to the PR.
* Refactor docs for inserting subtrees
This change is unrelated to the PR.
* Handle negative and zero getnetworksolsps arguments correctly
* Simplify the solsps loop structure
* Simplify loop by avoiding manual iteration and peeking
* Avoid division by zero
* Use min and max times rather than first and last times
* Refactor block iterators so they are more efficient
* Make finding chains easier
* Simplify block iteration code
* Remove implemented TODO comments
* Simplify internal iterator state
* Implement iteration by any chain item
* Iterate block headers rather than full blocks
* Ignore code that is (sometimes) dead
* Fix a dead code warning
* Add a no blocks in state error constant
* Check result values in the RPC test
* Fix invalid calculation handling
* add `sapling_subtree_for_tip` and `orchard_subtree_for_tip` methods to `ZebraDb`
* add methods for non finalized state, move functions
* call `zs_last_key_value` the right way
* fix and simplify `*_subtree_for_tip` methods
Co-authored-by: Arya <aryasolhi@gmail.com>
* apply filter
* rename all tree and subtree methods that use tip
* rename tip tree and subtree methods in non finalized chain
* apply simplify suggestions
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Provide a user hint when the database format is invalid
* Split a path-based database version method
* Check the database format before Zebra shuts down
* Fix a typo in zebra-state/src/service/finalized_state/zebra_db.rs
Co-authored-by: Marek <mail@marek.onl>
* Fix a typo in upgrade.rs
---------
Co-authored-by: Marek <mail@marek.onl>
* Refactor format checks, stop running quick check before upgrade
* Speed up startup by always running the format checks in parallel
* Make detailed format checks cancellable on shutdown
* Prepare for periodic format checks
* Time upgrades and validity checks
* Run a database format check every 5 minutes
* Wait for a cancel signal rather than unconditionally sleeping
* Move check_max_on_disk_height() into the format checks
* Move spawn_format_change() into its own method
* Avoid manual handling of previous sapling trees by using iterator windows instead
* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead
* Simplify adding notes by using the exact number of remaining notes
* Simplify by skipping the first block, because it can't complete a subtree
* Re-use existing tree update code
* Apply the sapling changes to orchard subtree updates
* add a reverse database column family iterator function
* Make skipping the lowest tree independent of iteration order
* Move new subtree checks into the iterator, rename to end_height
* Split subtree calculation into a new method
* Split the calculate and write methods
* Quickly check the first subtree before running the full upgrade
* Do the quick checks every time Zebra runs, and refactor slow check error handling
* Do quick checks for orchard as well
* Make orchard tree upgrade match sapling upgrade code
* Upgrade subtrees in reverse height order
* Bump the database patch version so the upgrade runs again
* Reset previous subtree upgrade data before doing this one
* Add extra checks to subtree calculation to diagnose errors
* Use correct heights for subtrees completed at the end of a block
* Add even more checks to diagnose issues
* Instrument upgrade methods to improve diagnostics
* Prevent modification of re-used trees
* Debug with subtree positions as well
* Fix an off-by-one error with completed subtrees
* Fix typos and confusing comments
Co-authored-by: Marek <mail@marek.onl>
* Fix mistaken previous tree handling and end tree comments
* Remove unnecessary subtraction in remaining leaves calc
* Log heights when assertions fail
* Fix new subtree detection filter
* Move new subtree check into a method, cleanup unused code
* Remove redundant assertions
* Wait for subtree upgrade before testing RPCs
* Fix subtree search in quick check
* Temporarily upgrade subtrees in forward height order
* Clarify some comments
* Fix missing test imports
* Fix subtree logging
* Add a comment about a potential hang with future upgrades
* Fix zebrad var ownership
* Log more info when add_subtrees.rs fails
* cargo fmt --all
* Fix unrelated clippy::unnecessary_unwrap
* cargo clippy --fix --all-features --all-targets; cargo fmt --all
* Stop the quick check depending on tree de-duplication
* Refactor waiting for the upgrade into functions
* Wait for state upgrades whenever the cached state is updated
* Wait for the testnet upgrade in the right place
* Fix unused variable
* Fix a subtree detection bug and comments
* Remove an early reference to reverse direction
* Stop skipping subtrees completed at the end of blocks
* Actually fix new subtree code
* Upgrade subtrees in reverse height order
Reverts "Temporarily upgrade subtrees in forward height order"
This reverts commit a9558be214.
* Bump the database patch version to re-run the upgrade (for testing)
* Revert "Remove an early reference to reverse direction"
This reverts commit c206404377.
---------
Co-authored-by: Marek <mail@marek.onl>
* Avoid manual handling of previous sapling trees by using iterator windows instead
* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead
* Simplify adding notes by using the exact number of remaining notes
* Simplify by skipping the first block, because it can't complete a subtree
* Re-use existing tree update code
* Apply the sapling changes to orchard subtree updates
* add a reverse database column family iterator function
* Make skipping the lowest tree independent of iteration order
* Move new subtree checks into the iterator, rename to end_height
* Split subtree calculation into a new method
* Split the calculate and write methods
* Quickly check the first subtree before running the full upgrade
* Do the quick checks every time Zebra runs, and refactor slow check error handling
* Do quick checks for orchard as well
* Make orchard tree upgrade match sapling upgrade code
* Upgrade subtrees in reverse height order
* Bump the database patch version so the upgrade runs again
* Reset previous subtree upgrade data before doing this one
* Add extra checks to subtree calculation to diagnose errors
* Use correct heights for subtrees completed at the end of a block
* Add even more checks to diagnose issues
* Instrument upgrade methods to improve diagnostics
* Prevent modification of re-used trees
* Debug with subtree positions as well
* Fix an off-by-one error with completed subtrees
* Fix typos and confusing comments
Co-authored-by: Marek <mail@marek.onl>
* Fix mistaken previous tree handling and end tree comments
* Remove unnecessary subtraction in remaining leaves calc
* Log heights when assertions fail
* Fix new subtree detection filter
* Move new subtree check into a method, cleanup unused code
* Remove redundant assertions
* Wait for subtree upgrade before testing RPCs
* Fix subtree search in quick check
* Temporarily upgrade subtrees in forward height order
* Clarify some comments
* Fix missing test imports
* Fix subtree logging
* Add a comment about a potential hang with future upgrades
* Fix zebrad var ownership
* Log more info when add_subtrees.rs fails
* cargo fmt --all
* Fix unrelated clippy::unnecessary_unwrap
* cargo clippy --fix --all-features --all-targets; cargo fmt --all
* Stop the quick check depending on tree de-duplication
* Refactor waiting for the upgrade into functions
* Wait for state upgrades whenever the cached state is updated
* Wait for the testnet upgrade in the right place
* Fix unused variable
* Fix a subtree detection bug and comments
* Remove an early reference to reverse direction
* Stop skipping subtrees completed at the end of blocks
* Actually fix new subtree code
---------
Co-authored-by: Marek <mail@marek.onl>
* Refactor some comments for subtrees
* Update docs about single subtree APIs
* Fix method rename in tree.rs
---------
Co-authored-by: teor <teor@riseup.net>
* Copy the add_subtrees upgrade from the original branch
* Copy the database write changes in shielded.rs from the original branch
* Copy the tree API changes from the original branch
* Simplify subtree APIs to avoid exposing frontiers
* Fix a dead code warning by re-using existing methods
* Use mpsc::Receiver<CancelFormatChange> in the subtree upgrade
* Run the subtree upgrade on startup
* Bump the database format version to 25.2.0
* Fix a confusing 'upgrade complete' log
* Clarify some comments and error messages
* Simplify prev_tree unwrap to avoid an (impossible?) concurrency bug
* Use separate subtree writing functions
* Use common note commitment list code
* Fix subtree completion condition and add asserts
* Simplify subtree API and avoid exposing Address
* Fix API compatibility when Arcs are removed
* Log when each subtree is added
* If a format change is cancelled, don't mark the database as upgraded or do format checks
* Log subtree progress about once every two minutes
* Adds a state validity check for subtrees upgrade
* Orchard is faster, decrease log interval
* Clarify subtree index docs
* Move a log to the correct location
* Refactor subtree upgrade to remove duplicate inverted loop conditions
* updates subtree state validity check
* Add a subtree format check when there is no upgrade
* Fix an off-by-one error with the final subtree check
* Use error-level logs for database format checks
* Skip format checks in tests that create invalid formats
* fix state validity test
* Add a concurrency comment to subtree by height methods
* Add individual subtree state methods: reverts removing these methods in an earlier PR
* fastmod "subtrees_by_index" "subtree_list_by_index_for_rpc"
---------
Co-authored-by: arya2 <aryasolhi@gmail.com>
* Make NoteCommitmentSubtreeIndex compatible with serde-based RPCs
* Add a stub for z_getsubtreesbyindex
* Define a GetSubtrees RPC response type
* Reject invalid shielded pool names
* Make limit optional
* Define state request and response types for subtrees
* Implement FromDisk for NoteCommitmentSubtreeIndex and add a round-trip test
* Make subtrees compatible with round-trip proptests
* Add finalized state subtree list methods and delete unused methods
* Remove Arc from subtrees in zebra-chain
* Remove Arc from subtrees in zebra-state and use BTreeMap
* Implement subtree list lookups in the non-finalized state and delete unused methods
* Implement consistent concurrent subtree read requests
* Implement ToHex for sapling::Node
* Implement ToHex for orchard::Node
* Implement z_get_subtrees_by_index RPC
* Check for the start_index from the non-finalized state
* Remove an unused mut
* Fix missing doc links
* Fix RPC comments
* Temporarily remove the z_get_subtrees_by_index RPC method
* Log errors and panic if duplicate trees are found after the de-duplicate upgrade
* Always check for duplicates, even if the state is already marked as upgraded
* Minor doc fixes
* Document ranges for `zs_delete_range`
* Revert the comment for `sapling_tree`
* Rearrange tree methods & fix their docs
* Bump DATABASE_FORMAT_PATCH_VERSION from 0 to 1
* Remove the manual tree deletion at early heights
* Add `skip_while` to `zs_range_iter`
* Refactor the tree deduplication
* Add comments to the pruning
* Turn warnings into panics
* Remove redundant checks
These checks are superseded by `check_for_duplicate_trees`
* Remove an edge case that ignored the last tree
* Suggestion for Fix the deduplication of note commitment trees (#7391)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Alfredo Garcia
Elijah Hampton
Marek
idky137
Za Wilcox
bishopcheckmate
dependabot[bot]
teor
Arya