* add `sapling_subtree_for_tip` and `orchard_subtree_for_tip` methods to `ZebraDb`
* add methods for non finalized state, move functions
* call `zs_last_key_value` the right way
* fix and simplify `*_subtree_for_tip` methods
Co-authored-by: Arya <aryasolhi@gmail.com>
* apply filter
* rename all tree and subtree methods that use tip
* rename tip tree and subtree methods in non finalized chain
* apply simplify suggestions
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Refactor format checks, stop running quick check before upgrade
* Speed up startup by always running the format checks in parallel
* Make detailed format checks cancellable on shutdown
* Prepare for periodic format checks
* Time upgrades and validity checks
* Run a database format check every 5 minutes
* Wait for a cancel signal rather than unconditionally sleeping
* Move check_max_on_disk_height() into the format checks
* Move spawn_format_change() into its own method
* Avoid manual handling of previous sapling trees by using iterator windows instead
* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead
* Simplify adding notes by using the exact number of remaining notes
* Simplify by skipping the first block, because it can't complete a subtree
* Re-use existing tree update code
* Apply the sapling changes to orchard subtree updates
* add a reverse database column family iterator function
* Make skipping the lowest tree independent of iteration order
* Move new subtree checks into the iterator, rename to end_height
* Split subtree calculation into a new method
* Split the calculate and write methods
* Quickly check the first subtree before running the full upgrade
* Do the quick checks every time Zebra runs, and refactor slow check error handling
* Do quick checks for orchard as well
* Make orchard tree upgrade match sapling upgrade code
* Upgrade subtrees in reverse height order
* Bump the database patch version so the upgrade runs again
* Reset previous subtree upgrade data before doing this one
* Add extra checks to subtree calculation to diagnose errors
* Use correct heights for subtrees completed at the end of a block
* Add even more checks to diagnose issues
* Instrument upgrade methods to improve diagnostics
* Prevent modification of re-used trees
* Debug with subtree positions as well
* Fix an off-by-one error with completed subtrees
* Fix typos and confusing comments
Co-authored-by: Marek <mail@marek.onl>
* Fix mistaken previous tree handling and end tree comments
* Remove unnecessary subtraction in remaining leaves calc
* Log heights when assertions fail
* Fix new subtree detection filter
* Move new subtree check into a method, cleanup unused code
* Remove redundant assertions
* Wait for subtree upgrade before testing RPCs
* Fix subtree search in quick check
* Temporarily upgrade subtrees in forward height order
* Clarify some comments
* Fix missing test imports
* Fix subtree logging
* Add a comment about a potential hang with future upgrades
* Fix zebrad var ownership
* Log more info when add_subtrees.rs fails
* cargo fmt --all
* Fix unrelated clippy::unnecessary_unwrap
* cargo clippy --fix --all-features --all-targets; cargo fmt --all
* Stop the quick check depending on tree de-duplication
* Refactor waiting for the upgrade into functions
* Wait for state upgrades whenever the cached state is updated
* Wait for the testnet upgrade in the right place
* Fix unused variable
* Fix a subtree detection bug and comments
* Remove an early reference to reverse direction
* Stop skipping subtrees completed at the end of blocks
* Actually fix new subtree code
* Upgrade subtrees in reverse height order
Reverts "Temporarily upgrade subtrees in forward height order"
This reverts commit a9558be214.
* Bump the database patch version to re-run the upgrade (for testing)
* Revert "Remove an early reference to reverse direction"
This reverts commit c206404377.
---------
Co-authored-by: Marek <mail@marek.onl>
* Avoid manual handling of previous sapling trees by using iterator windows instead
* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead
* Simplify adding notes by using the exact number of remaining notes
* Simplify by skipping the first block, because it can't complete a subtree
* Re-use existing tree update code
* Apply the sapling changes to orchard subtree updates
* add a reverse database column family iterator function
* Make skipping the lowest tree independent of iteration order
* Move new subtree checks into the iterator, rename to end_height
* Split subtree calculation into a new method
* Split the calculate and write methods
* Quickly check the first subtree before running the full upgrade
* Do the quick checks every time Zebra runs, and refactor slow check error handling
* Do quick checks for orchard as well
* Make orchard tree upgrade match sapling upgrade code
* Upgrade subtrees in reverse height order
* Bump the database patch version so the upgrade runs again
* Reset previous subtree upgrade data before doing this one
* Add extra checks to subtree calculation to diagnose errors
* Use correct heights for subtrees completed at the end of a block
* Add even more checks to diagnose issues
* Instrument upgrade methods to improve diagnostics
* Prevent modification of re-used trees
* Debug with subtree positions as well
* Fix an off-by-one error with completed subtrees
* Fix typos and confusing comments
Co-authored-by: Marek <mail@marek.onl>
* Fix mistaken previous tree handling and end tree comments
* Remove unnecessary subtraction in remaining leaves calc
* Log heights when assertions fail
* Fix new subtree detection filter
* Move new subtree check into a method, cleanup unused code
* Remove redundant assertions
* Wait for subtree upgrade before testing RPCs
* Fix subtree search in quick check
* Temporarily upgrade subtrees in forward height order
* Clarify some comments
* Fix missing test imports
* Fix subtree logging
* Add a comment about a potential hang with future upgrades
* Fix zebrad var ownership
* Log more info when add_subtrees.rs fails
* cargo fmt --all
* Fix unrelated clippy::unnecessary_unwrap
* cargo clippy --fix --all-features --all-targets; cargo fmt --all
* Stop the quick check depending on tree de-duplication
* Refactor waiting for the upgrade into functions
* Wait for state upgrades whenever the cached state is updated
* Wait for the testnet upgrade in the right place
* Fix unused variable
* Fix a subtree detection bug and comments
* Remove an early reference to reverse direction
* Stop skipping subtrees completed at the end of blocks
* Actually fix new subtree code
---------
Co-authored-by: Marek <mail@marek.onl>
* Copy the add_subtrees upgrade from the original branch
* Copy the database write changes in shielded.rs from the original branch
* Copy the tree API changes from the original branch
* Simplify subtree APIs to avoid exposing frontiers
* Fix a dead code warning by re-using existing methods
* Use mpsc::Receiver<CancelFormatChange> in the subtree upgrade
* Run the subtree upgrade on startup
* Bump the database format version to 25.2.0
* Fix a confusing 'upgrade complete' log
* Clarify some comments and error messages
* Simplify prev_tree unwrap to avoid an (impossible?) concurrency bug
* Use separate subtree writing functions
* Use common note commitment list code
* Fix subtree completion condition and add asserts
* Simplify subtree API and avoid exposing Address
* Fix API compatibility when Arcs are removed
* Log when each subtree is added
* If a format change is cancelled, don't mark the database as upgraded or do format checks
* Log subtree progress about once every two minutes
* Adds a state validity check for subtrees upgrade
* Orchard is faster, decrease log interval
* Clarify subtree index docs
* Move a log to the correct location
* Refactor subtree upgrade to remove duplicate inverted loop conditions
* updates subtree state validity check
* Add a subtree format check when there is no upgrade
* Fix an off-by-one error with the final subtree check
* Use error-level logs for database format checks
* Skip format checks in tests that create invalid formats
* fix state validity test
* Add a concurrency comment to subtree by height methods
* Add individual subtree state methods: reverts removing these methods in an earlier PR
* fastmod "subtrees_by_index" "subtree_list_by_index_for_rpc"
---------
Co-authored-by: arya2 <aryasolhi@gmail.com>
Marek
Alfredo Garcia
teor
Arya