* impl TryFrom<zcash_primitives::BlockHeight> for Height
* Add type-safe read and write database methods
* Only allow typed access to the scanner DB
* Update docs
* Implement a common method as a trait
* Fix imports
* Tidy state imports
* Activate tracing logging macros in the whole scanner crate
* Fix dead code warnings
* Use a more sensible export order
* Remove a 1.72 lint exception now 1.74 is stable
* Switch history trees over to TypedColumnFamily, and remove redundant code
* Add typed batch creation methods, and switch history trees to them
* Convert ValueBalance to typed column families
* Make the APIs compatible after a merge
* Use `ZebraDb` instead of `DiskDb` where needed
---------
Co-authored-by: Marek <mail@marek.onl>
* impl TryFrom<zcash_primitives::BlockHeight> for Height
* Add type-safe read and write database methods
* Only allow typed access to the scanner DB
* Update docs
* Implement a common method as a trait
* Fix imports
* Tidy state imports
* Activate tracing logging macros in the whole scanner crate
* Fix dead code warnings
* start scanner where it was left
* fix tests
* add a `scan_start_where_left` test
* refactor a log msg
* fix some comments
* remove function
* fix doc comment
* clippy
* fix `sapling_keys_and_last_scanned_heights()`
* simplify start height
* i went too far, revert some changes back
* change log info to every 10k blocks
* fix build
* Update height snapshot code and check last height is consistent
* Add strictly before and strictly after database key gets
* Move to the previous key using strictly before ops
* Assert that keys are only inserted once
* Update the index in each loop
* Update snapshots
* Remove debugging code
* start scanning at min available height
---------
Co-authored-by: teor <teor@riseup.net>
* Refactor test data into functions
* Add a typed snapshot test for scanner storage
* Use standard hex serialization with SaplingScannedResult
* Simplify transaction::Hash hex serialization
* Sort HashMaps before snapshotting
* Add typed snapshot data
* Make some scanner storage methods more flexible
* Move tests to a submodule and expose test functions and constants
* Make scanner functions clearer and easier to use
* Simplify state snapshot test code
* Add raw data snapshot tests for the scanner
* Add snapshots
* Fix import path
* Fix import conditional compilation
* fix imports
* fix imports 2
* Put read and write db exports together
* Remove confusing IntoDisk/FromDisk impl
* Fix an incorrect unused method that could panic
* Delete a test that is no longer valid
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Upgrade the scanner database major version to 1
* Update format docs
* Change the high-level scanner db format
* Change the scanner serialization formats
* Fix value format and tests
* Fix incorrect types
* Update documentation
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Fix availability of tokio::time in scanner
* Create a function that parses a key into a list of keys
* Pass a ChainTipChange to the scanner function
* Convert a scanned block to a sapling result
* Make it easier to pass keys and blocks
* Increase scanner wait times
* Parse keys once at the start of the scan
* Get a block from the state instead of the tip
* Don't log secret keys, only log every 100,000 blocks
* Scan each block and add the results to storage
* Move blocking tasks into spawn_blocking()
* Update the acceptance test
* Use a dummy sapling tree size
* Use a larger dummy size
* Expose IntoDisk and FromDisk in zebra-state
* Implement database serialization for SaplingScanningKey Strings
* Implement serialization for Vec<SaplingScannedResult> (Vec<transaction::Hash>)
* Implement seralization for SaplingScannedDatabaseIndex
* Add an is_empty() method
* Add a read method for a specific index, and document it
* Implement writing scanner results to the database
* Make read name more explicit
* Implement writing scanner keys
* Implement reading sapling keys
* Spawn blocking tasks correctly in async code
* Change storage results methods to use the database
* Update tests that use storage
* Use spawn_blocking() for database methods
* Change the check interval to slightly less than the block interval
* Expose raw database methods with shielded-scan
* fix `scan_task_starts` test
* minor doc change in test
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Create an empty storage/db module
* Use ephemeral storage in tests
* Populate storage inside new() method
* Move scanner setup into an init() method
* Pass the network to scanner init
* Create a database but don't actually use it
* Skip shutdown format checks when skipping format upgrades
* Allow the scanner to skip launching format upgrades in production
* Refactor skipping format upgrades so it is consistent
* Allow checking configs for equality
* Restore Network import
* Provide access to DiskDb and DiskWriteBatch outside the state using a feature
* Actually let's export ZebraDb for the format upgrade code
* Pass column families to ZebraDb as an argument
* Allow the database kind to be changed in config.rs
* Use the state kind in finalized_state.rs
* Allow different database kinds in ZebraDb, but don't move the upgrade code yet
* Allow different database kinds in DiskDb
* Allow different database kinds in upgrade.rs, but don't split the upgrade code out yet
* Add new arguments to raw database tests
* Fix doc links
* Fix internal imports
* Fix unused code
* Update zebrad version metadata
* Create a specific state database delete function
* Fix state exports
* Fix zebrad tests
* Fix zebrad state write tests
* Make CI run again
* Fix dead code warnings for test methods
* Remove unnecessary async on some tests
* Fix logging required by tests
* Fix logging required in test itself
* Fix variable names
* Try to copy the message and add regexes
* Avoid a race condition in the 25.3.0 state upgrade
* Look for the most recent sprout tree height if needed
* Get the latest history tree not the tip height history tree
* Discard keys provided by low level database method
* Remove extra ;
* Provide key types & rustfmt
* Fix weird closure type syntax
* Update zebra-state/src/service/finalized_state/disk_format/upgrade/fix_tree_key_type.rs
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* Refactor return type of poll_discover()
* Simplify poll_ready() by removing preselected peers
* Fix peer set readiness check
* Pass task context correctly to background tasks
* Make poll_discover() return Pending
* Make poll_inventory() return Pending
* Make poll_unready() return Poll::Pending
* Simplify with futures::ready!() and ?
* When there are no peers, wake on newly ready peers, or new peers, in that order
* Preserve the original waker when there are no unready peers
* Fix polling docs and remove unnecessary code
* Make sure we're ignoring Poll::Pending not Result
* Make panic checking method names clearer
* Fix connection Client task wakeups and error handling
* Cleanup connection panic handling and add wakeup docs
* Fix connection client task wakeups to prevent hangs
* Simplify error and pending handling
* Clarify inventory set behaviour
* Define peer set poll_* methods so they return Ok if they do something
* Clarify documentation
Co-authored-by: Arya <aryasolhi@gmail.com>
* Fix test that depended on preselected peers
* Check ready peers for errors before sending requests to them
* Fix a hanging test by not waiting for irrelevant actions
* Only remove cancel handles when they are required
* fix incorrect panic on termination setting
* Clarify method comments
Co-authored-by: Arya <aryasolhi@gmail.com>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Stop using `SemanticallyVerifiedBlockWithTrees`
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Doc that default treestate corresponds to genesis
* Doc the diff between default values of tree roots
---------
Co-authored-by: teor <teor@riseup.net>
* Simple replacements of doc.zebra.zfnd.org with docs.rs
* Manual fixes for specific main/internal/external docs
* Point developer docs to doc-internal.zebra.zfnd.org
* fastmod --glob '\!.git' -- doc.zebra.zfnd.org/zebrad docs.rs/zebrad/latest/zebrad
* Manually remove any remaining doc.zfnd.zebra.org links
* Remove the external docs job
* Add changelog entry and fix links
* Fix links that were broken before this PR
* refactors sapling_subtrees to use ranges
* refactor orchard_subtrees() to use ranges
* Restores check for inconsistent subtree roots
* updates vectors tests
* Applies some suggestions from code review and adds `read::tree::subtrees` function
* updates correctness comment & db read method names
* adds test_subtrees
* tests in-memory reads
* test that subtree read fns work right with excluded start bound
* applies suggestions from code review
* Applies suggestions from code review.
* Updates docs applying suggestion from code review
* adds new arg to zs_range_iter calls
* Add tree key format and cached root upgrades
* Document the changes in the upgrades
* Remove unnecessary clippy::unwrap_in_result
* Fix database type
* Bump state version
* Skip some checks if the database is empty
* Fix tests for a short state upgrade
* Disable format checks in some tests
* Document state performance issues
* Clarify upgrade behaviour
* Clarify panic messages
* Delete incorrect genesis trees write code
* Fix metrics handling for genesis
* Remove an unused import
* Explain why genesis anchors are ok
* Update snapshots
* Debug a failing test
* Fix some tests
* Fix missing imports
* Move the state check in a test
* Fix comment and doc typos
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
* Clarify what a long upgrade is
* Rename unused function arguments
Co-authored-by: Marek <mail@marek.onl>
* Add all_unordered log regex matching methods
* Fix timing issues with version upgrades and other logs
* Fix argument name in docs
Co-authored-by: Marek <mail@marek.onl>
* Explain match until first for all regexes behaviour better
---------
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
* Uses range_iter in address_transaction_locations
* Uses range_iter in address_transaction_locations
* uses u16::MAX instead of usize::MAX
* Moves limit code into method
* adds allow(dead_code)
* Simplifies address_iterator_range
* Moves test state init out of loop
* Updates docs
* Fix the combining of Sapling subtrees
This commit fixes a bug due to which the function `sapling_subtrees`
used to always include all subtrees from the non-finalized state without
respecting the given limit.
* Fix the combining of Orchard subtrees
This commit fixes a bug due to which the function `orchard_subtrees`
used to always include all subtrees from the non-finalized state without
respecting the given limit.
* Add additional checks when retrieving subtrees
* Allow raw subtree insertions into `Chain` in tests
* Derive `Default` for `Chain`
* Derive `Default` for note commitment tree nodes
* Use `pub(super)` for `DiskWriteBatch`
Set the visibility of `DiskWriteBatch` to `pub(super)`.
* Add tests for retrieving subtrees
* Use `default()` for subtree roots in tests
This change is unrelated to the PR.
* Refactor docs for inserting subtrees
This change is unrelated to the PR.
* Handle negative and zero getnetworksolsps arguments correctly
* Simplify the solsps loop structure
* Simplify loop by avoiding manual iteration and peeking
* Avoid division by zero
* Use min and max times rather than first and last times
* Refactor block iterators so they are more efficient
* Make finding chains easier
* Simplify block iteration code
* Remove implemented TODO comments
* Simplify internal iterator state
* Implement iteration by any chain item
* Iterate block headers rather than full blocks
* Ignore code that is (sometimes) dead
* Fix a dead code warning
* Add a no blocks in state error constant
* Check result values in the RPC test
* Fix invalid calculation handling
* add `sapling_subtree_for_tip` and `orchard_subtree_for_tip` methods to `ZebraDb`
* add methods for non finalized state, move functions
* call `zs_last_key_value` the right way
* fix and simplify `*_subtree_for_tip` methods
Co-authored-by: Arya <aryasolhi@gmail.com>
* apply filter
* rename all tree and subtree methods that use tip
* rename tip tree and subtree methods in non finalized chain
* apply simplify suggestions
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Provide a user hint when the database format is invalid
* Split a path-based database version method
* Check the database format before Zebra shuts down
* Fix a typo in zebra-state/src/service/finalized_state/zebra_db.rs
Co-authored-by: Marek <mail@marek.onl>
* Fix a typo in upgrade.rs
---------
Co-authored-by: Marek <mail@marek.onl>
* Refactor format checks, stop running quick check before upgrade
* Speed up startup by always running the format checks in parallel
* Make detailed format checks cancellable on shutdown
* Prepare for periodic format checks
* Time upgrades and validity checks
* Run a database format check every 5 minutes
* Wait for a cancel signal rather than unconditionally sleeping
* Move check_max_on_disk_height() into the format checks
* Move spawn_format_change() into its own method
* Avoid manual handling of previous sapling trees by using iterator windows instead
* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead
* Simplify adding notes by using the exact number of remaining notes
* Simplify by skipping the first block, because it can't complete a subtree
* Re-use existing tree update code
* Apply the sapling changes to orchard subtree updates
* add a reverse database column family iterator function
* Make skipping the lowest tree independent of iteration order
* Move new subtree checks into the iterator, rename to end_height
* Split subtree calculation into a new method
* Split the calculate and write methods
* Quickly check the first subtree before running the full upgrade
* Do the quick checks every time Zebra runs, and refactor slow check error handling
* Do quick checks for orchard as well
* Make orchard tree upgrade match sapling upgrade code
* Upgrade subtrees in reverse height order
* Bump the database patch version so the upgrade runs again
* Reset previous subtree upgrade data before doing this one
* Add extra checks to subtree calculation to diagnose errors
* Use correct heights for subtrees completed at the end of a block
* Add even more checks to diagnose issues
* Instrument upgrade methods to improve diagnostics
* Prevent modification of re-used trees
* Debug with subtree positions as well
* Fix an off-by-one error with completed subtrees
* Fix typos and confusing comments
Co-authored-by: Marek <mail@marek.onl>
* Fix mistaken previous tree handling and end tree comments
* Remove unnecessary subtraction in remaining leaves calc
* Log heights when assertions fail
* Fix new subtree detection filter
* Move new subtree check into a method, cleanup unused code
* Remove redundant assertions
* Wait for subtree upgrade before testing RPCs
* Fix subtree search in quick check
* Temporarily upgrade subtrees in forward height order
* Clarify some comments
* Fix missing test imports
* Fix subtree logging
* Add a comment about a potential hang with future upgrades
* Fix zebrad var ownership
* Log more info when add_subtrees.rs fails
* cargo fmt --all
* Fix unrelated clippy::unnecessary_unwrap
* cargo clippy --fix --all-features --all-targets; cargo fmt --all
* Stop the quick check depending on tree de-duplication
* Refactor waiting for the upgrade into functions
* Wait for state upgrades whenever the cached state is updated
* Wait for the testnet upgrade in the right place
* Fix unused variable
* Fix a subtree detection bug and comments
* Remove an early reference to reverse direction
* Stop skipping subtrees completed at the end of blocks
* Actually fix new subtree code
* Upgrade subtrees in reverse height order
Reverts "Temporarily upgrade subtrees in forward height order"
This reverts commit a9558be214.
* Bump the database patch version to re-run the upgrade (for testing)
* Revert "Remove an early reference to reverse direction"
This reverts commit c206404377.
---------
Co-authored-by: Marek <mail@marek.onl>
* Avoid manual handling of previous sapling trees by using iterator windows instead
* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead
* Simplify adding notes by using the exact number of remaining notes
* Simplify by skipping the first block, because it can't complete a subtree
* Re-use existing tree update code
* Apply the sapling changes to orchard subtree updates
* add a reverse database column family iterator function
* Make skipping the lowest tree independent of iteration order
* Move new subtree checks into the iterator, rename to end_height
* Split subtree calculation into a new method
* Split the calculate and write methods
* Quickly check the first subtree before running the full upgrade
* Do the quick checks every time Zebra runs, and refactor slow check error handling
* Do quick checks for orchard as well
* Make orchard tree upgrade match sapling upgrade code
* Upgrade subtrees in reverse height order
* Bump the database patch version so the upgrade runs again
* Reset previous subtree upgrade data before doing this one
* Add extra checks to subtree calculation to diagnose errors
* Use correct heights for subtrees completed at the end of a block
* Add even more checks to diagnose issues
* Instrument upgrade methods to improve diagnostics
* Prevent modification of re-used trees
* Debug with subtree positions as well
* Fix an off-by-one error with completed subtrees
* Fix typos and confusing comments
Co-authored-by: Marek <mail@marek.onl>
* Fix mistaken previous tree handling and end tree comments
* Remove unnecessary subtraction in remaining leaves calc
* Log heights when assertions fail
* Fix new subtree detection filter
* Move new subtree check into a method, cleanup unused code
* Remove redundant assertions
* Wait for subtree upgrade before testing RPCs
* Fix subtree search in quick check
* Temporarily upgrade subtrees in forward height order
* Clarify some comments
* Fix missing test imports
* Fix subtree logging
* Add a comment about a potential hang with future upgrades
* Fix zebrad var ownership
* Log more info when add_subtrees.rs fails
* cargo fmt --all
* Fix unrelated clippy::unnecessary_unwrap
* cargo clippy --fix --all-features --all-targets; cargo fmt --all
* Stop the quick check depending on tree de-duplication
* Refactor waiting for the upgrade into functions
* Wait for state upgrades whenever the cached state is updated
* Wait for the testnet upgrade in the right place
* Fix unused variable
* Fix a subtree detection bug and comments
* Remove an early reference to reverse direction
* Stop skipping subtrees completed at the end of blocks
* Actually fix new subtree code
---------
Co-authored-by: Marek <mail@marek.onl>
* Refactor some comments for subtrees
* Update docs about single subtree APIs
* Fix method rename in tree.rs
---------
Co-authored-by: teor <teor@riseup.net>
* Copy the add_subtrees upgrade from the original branch
* Copy the database write changes in shielded.rs from the original branch
* Copy the tree API changes from the original branch
* Simplify subtree APIs to avoid exposing frontiers
* Fix a dead code warning by re-using existing methods
* Use mpsc::Receiver<CancelFormatChange> in the subtree upgrade
* Run the subtree upgrade on startup
* Bump the database format version to 25.2.0
* Fix a confusing 'upgrade complete' log
* Clarify some comments and error messages
* Simplify prev_tree unwrap to avoid an (impossible?) concurrency bug
* Use separate subtree writing functions
* Use common note commitment list code
* Fix subtree completion condition and add asserts
* Simplify subtree API and avoid exposing Address
* Fix API compatibility when Arcs are removed
* Log when each subtree is added
* If a format change is cancelled, don't mark the database as upgraded or do format checks
* Log subtree progress about once every two minutes
* Adds a state validity check for subtrees upgrade
* Orchard is faster, decrease log interval
* Clarify subtree index docs
* Move a log to the correct location
* Refactor subtree upgrade to remove duplicate inverted loop conditions
* updates subtree state validity check
* Add a subtree format check when there is no upgrade
* Fix an off-by-one error with the final subtree check
* Use error-level logs for database format checks
* Skip format checks in tests that create invalid formats
* fix state validity test
* Add a concurrency comment to subtree by height methods
* Add individual subtree state methods: reverts removing these methods in an earlier PR
* fastmod "subtrees_by_index" "subtree_list_by_index_for_rpc"
---------
Co-authored-by: arya2 <aryasolhi@gmail.com>
* Make NoteCommitmentSubtreeIndex compatible with serde-based RPCs
* Add a stub for z_getsubtreesbyindex
* Define a GetSubtrees RPC response type
* Reject invalid shielded pool names
* Make limit optional
* Define state request and response types for subtrees
* Implement FromDisk for NoteCommitmentSubtreeIndex and add a round-trip test
* Make subtrees compatible with round-trip proptests
* Add finalized state subtree list methods and delete unused methods
* Remove Arc from subtrees in zebra-chain
* Remove Arc from subtrees in zebra-state and use BTreeMap
* Implement subtree list lookups in the non-finalized state and delete unused methods
* Implement consistent concurrent subtree read requests
* Implement ToHex for sapling::Node
* Implement ToHex for orchard::Node
* Implement z_get_subtrees_by_index RPC
* Check for the start_index from the non-finalized state
* Remove an unused mut
* Fix missing doc links
* Fix RPC comments
* Temporarily remove the z_get_subtrees_by_index RPC method
* Log errors and panic if duplicate trees are found after the de-duplicate upgrade
* Always check for duplicates, even if the state is already marked as upgraded
* Minor doc fixes
* Document ranges for `zs_delete_range`
* Revert the comment for `sapling_tree`
* Rearrange tree methods & fix their docs
* Bump DATABASE_FORMAT_PATCH_VERSION from 0 to 1
* Remove the manual tree deletion at early heights
* Add `skip_while` to `zs_range_iter`
* Refactor the tree deduplication
* Add comments to the pruning
* Turn warnings into panics
* Remove redundant checks
These checks are superseded by `check_for_duplicate_trees`
* Remove an edge case that ignored the last tree
* Suggestion for Fix the deduplication of note commitment trees (#7391)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* If there's an existing database with no version file, give it version 25.0.0
* Creating the RocksDB database makes a temporary change to the default database version
* zebra-chain changes from the subtree-boundaries branch
```sh
git checkout -b subtree-boundaries-zebra-chain main
git checkout origin/subtree-boundaries zebra-chain
git commit
```
* Temporarily populate new subtree fields with None - for revert
This temporary commit needs to be reverted in the next PR.
* Applies suggestions from code review
* removes from_repr_unchecked methods
* simplifies loop
* adds subtrees to zebra-state
* uses split_at, from_repr, & updates state-db-upgrades.md
* Update book/src/dev/state-db-upgrades.md
Co-authored-by: teor <teor@riseup.net>
* renames partial_subtree to subtree_data
* tests that subtree serialization format
* adds raw data format serialization round-trip test
* decrements minor version and skips inserting subtrees in db
---------
Co-authored-by: teor <teor@riseup.net>
* zebra-chain changes from the subtree-boundaries branch
```sh
git checkout -b subtree-boundaries-zebra-chain main
git checkout origin/subtree-boundaries zebra-chain
git commit
```
* Temporarily populate new subtree fields with None - for revert
This temporary commit needs to be reverted in the next PR.
* Applies suggestions from code review
* removes from_repr_unchecked methods
* simplifies loop
---------
Co-authored-by: arya2 <aryasolhi@gmail.com>
* fix clippy and rustfmt after last releases today
* Ignore clippy::arc_with_non_send_sync false positives in proptest code only
* Remove .cargo/config.toml changes that didn't work
---------
Co-authored-by: teor <teor@riseup.net>
* Fix#6388, rename sent_hashes field
* Removes prune_by_height, uses new SentHashes instead
* update queue_and_commit_to_non_finalized_state to start with children of non-finalized tip when dropping the finalized block write sender
* revert rename for now
* removes outdated TODO
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add support for deleting the trees
* Prune the trees
* Remove `Network` from `DiskWriteBatch`
Removing the `Network` from `DiskWriteBatch` makes it easy to
instantiate `DiskWriteBatch`es in `ZebraDb` that remove individual note
commitment trees.
The `Network` from `DiskWriteBatch` was used only for transparent
addresses, so the refactor isn't large. After removing it from
`DiskWriteBatch`, I passed it as a function argument instead. However,
we should simplify the parameter lists because at least two functions
have more than seven parameters now.
* Support individual tree removal in `ZebraDb`
* Refactor the tree removal task
* Prune old comments
* Remove redundant code
* Batch the removals
* delete ranges before relevant network upgrades
* moves prev_tree inits
* add iterator methods for reading note commitment trees
* Sets up skeleton of sapling pipeline
* Replaces .filter with .take_while
Fills in pipeline
Reuses zs_range_iter instead of repeating that code
Updates logic to stop at initial tip height
* uses std threads
* delete_range excludes end key
* fixes off by one bugs
* Log warning when a send fails
* Removes progress logs
* Log join errors instead of panicking
* Revert: Make the `db` field of `ZebraDb` private
* Move `delete_range_sapling_tree`
* Remove a redundant `else if` branch
Rationale:
The condition `n == 1` for the removed branch is true for a subset of
values of `n` in the preceding condition `n >= 1`.
* Use more specific error messages
* Revert: Remove redundant methods for tree removal
* Suggestions for Deduplicate note commitment trees stored in the finalized state (#7330)
* Add TODOs to some `Height` methods
* Add methods for deleting individual trees
* Refactor the tasks for deleting trees
---------
Co-authored-by: arya2 <aryasolhi@gmail.com>
* Pass ZebraDB to batch preparation
* Dedup the insertion of Sapling trees into database
* Dedup the insertion of Orchard trees into database
* Update snapshots
* Rename batch preparation of trees
* Simplify the naming of note commitment trees
* Correctly retrieve Sapling trees from fin state
* Correctly retrieve Orchard trees from fin state
* Simplify the naming of methods for Sprout trees
* Simplify the naming of methods for Sapling trees
* Simplify the naming of methods for Orchard trees
* Reduce disk reads by caching trees. (#7276)
* Bump the state minor version
* Reset the state patch version
* Simplify the preparation of genesis trees
* Store the roots of the trees of the genesis block
* Add the genesis roots to snapshots
* fix(test): Don't include shielded data in genesis blocks (#7302)
* fix(state): Fix marking format upgrades (#7304)
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Move the state format into a new doc
* Add upgrade instructions
* Link to the format upgrade docs from the upgrade code
* Fix typo
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* Remove duplicate asserts
There are the same two asserts above the two removed ones.
* Remove workarounds for inserting trees into NFS
NFS = non finalized state
* Use correct height for constructing new chain
We were using the height of the last block instead of the initial block
to construct a new chain.
* Don't push the 0th block into a chain
* Don't commit two blocks at the same height
* Add helpers for heights
* Support the retrieval of deduped Sprout trees
* Dedup Sprout trees
* Refactor docs for adding & removing Sprout trees
* Support the retrieval of deduped Sapling trees
* Dedup Sapling trees
* Refactor docs for adding & removing Sapling trees
* Support the retrieval of deduped Orchard trees
* Dedup Orchard trees
* Refactor docs for adding & removing Orchard trees
* Make the docs for storing trees clearer
* Remove duplicate asserts
There are the same two asserts above the two removed ones.
* Remove workarounds for inserting trees into NFS
NFS = non finalized state
* Use correct height for constructing new chain
We were using the height of the last block instead of the initial block
to construct a new chain.
* Don't push the 0th block into a chain
* Don't commit two blocks at the same height
* Fix typo
* Generate chains with at least two blocks
---------
Co-authored-by: teor <teor@riseup.net>
* Add an async-error feature and an initial module structure
* Implement checking for panics in OS threads and async tasks
* Implement waiting for panics in OS threads and async tasks
* Add a TODO to simplify some state request error handling
* Use the new panic-checking methods in zebra-state
* Use new panic-checking methods in zebra-network
* fixup! Implement waiting for panics in OS threads and async tasks
* Replace existing async code with generic panic-checking methods
* Simplify trait to a single method
* Move thread panic code into generic trait impls
* Simplify option handling
Co-authored-by: Arya <aryasolhi@gmail.com>
* Fix comment
Co-authored-by: Arya <aryasolhi@gmail.com>
* Add missing track_caller
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* update ecc dependencies no serialization
* use zcash_primitives merke tree functions for serialization/deserialization in orchard and sapling trees
* some more work in sapling/orchard serialization/deserialization
* clippy
* fix doc links
* fix missing doc
* make orchard trees serializa/deserialize as they were before upgrade
* make sapling trees serialize/deserialize as they were before upgrade
* use legacy for sprout
* remove unused code
* readd snapshot tests
* repalce some code
* upgrade zcash_proofs
* remove legacy code for sprout
* fix the count method
* add root to serialize
* fixes in as_bytes and from_bytes
* use legacy code
* add todo about pow2 tests
* remove unused sprout code
* fix doc typos
* Add a recalculate_root() method to trees for tests
* Rename test tree types to make them easier to change
* Add TODOs for tests for old and new serialization formats
* fix doc typos
* add more test to note commitment trees
* fix comment
* fix leaf serializatiuon, reverse to old hashes in pow2 tests
* fix serialization
* put sapling SerializedTree code back
* put orchard SerializedTree code back
* clippy
* add duplicated dependencies until zebra_script updates
* fix a doc link
* minor cleanup
* remove todo comment from tests
* add one more check to tests
* update zebra_script
* update deny.toml
* replace custom function with library
* fix some tests
* update docs
* Remove duplicate dependencies from deny.toml
Based on the list at:
https://github.com/ZcashFoundation/zebra/actions/runs/5557139662/jobs/10150543673?pr=7053#step:5:14
---------
Co-authored-by: teor <teor@riseup.net>
* Move format upgrades to their own module and enum
* Launch a format change thread if needed, and shut it down during shutdown
* Add some TODOs and remove a redundant timer
* Regularly check for panics in the state upgrade task
* Only run example upgrade once, change version field names
* Increment database format to 25.0.2: add format change task
* Log the running and initial disk database format versions on startup
* Add initial disk and running state versions to cached state images in CI
* Fix missing imports
* Fix typo in logs workflow command
* Add a force_save_to_disk argument to the CI workflow
* Move use_internet_connection into zebrad_config()
* fastmod can_spawn_zebrad_for_rpc can_spawn_zebrad_for_test_type zebra*
* Add a spawn_zebrad_without_rpc() function
* Remove unused copy_state() test code
* Assert that upgrades and downgrades happen with the correct versions
* Add a kill_and_return_output() method for tests
* Add a test for new_state_format() versions (no upgrades or downgrades)
* Add use_internet_connection to can_spawn_zebrad_for_test_type()
* Fix workflow parameter passing
* Check that reopening a new database doesn't upgrade (or downgrade) the format
* Allow ephemeral to be set to false even if we don't have a cached state
* Add a test type that will accept any kind of state
* When re-using a directory, configure the state test config with that path
* Actually mark newly created databases with their format versions
* Wait for the state to be opened before testing the format
* Run state format tests on mainnet and testnet configs (no network access)
* run multiple reopens in tests
* Test upgrades run correctly
* Test that version downgrades work as expected (best effort)
* Add a TODO for testing partial updates
* Fix missing test arguments
* clippy if chain
* Fix typo
* another typo
* Pass a database instance to the format upgrade task
* Fix a timing issue in the tests
* Fix version matching in CI
* Use correct env var reference
* Use correct github env file
* Wait for the database to be written before killing Zebra
* Use correct workflow syntax
* Version changes aren't always upgrades
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a recalculate_root() method to trees for tests
* Rename test tree types to make them easier to change
* Add TODOs for tests for old and new serialization formats
* fix doc typos
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Add an assert_frontier_eq() method to note commitment trees for tests
* Check round-trip serialization for note commitment trees
* fix typos
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove duplicate "running" message, send log file opening logs to stderr
* Add missing block progress bar desc, make them consistent
* Put progress bars in a specific order in each section
* Actually make block progress bar desc consistent
* Make progress bar order even more consistent
* Fix fork blocks plural for 1 block
* Use the correct number of chain fork bars
* Disable confusing partial work display
* Add struct field category comments
* Silence a verbose inventory log
* claridy some checkpoint verifier docs
* update documentation of `Request::CommitSemanticallyVerifiedBlock` and `Request::CommitCheckpointVerifiedBlock`
* replace `prepared` with `semantically_verified` in state service checks code
* replace `non-finalized` where needed in docs of the state service
* fix double space in doc
* replace `finalized` where needed in docs of the state service
* change some docs in state queued_blocks.rs
* Rewrite pending UTXO checkpoint block comment
* Fix trailing space in docs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: teor <teor@riseup.net>
* Activate production features on docs.rs
* Make version requirements more flexible to work around unpublished versions
* Remove a redundant feature requirement that causes publishing issues
* fix release auto-replacement format
* cargo release replace --verbose --execute --package zebrad
* Add a missing 1.0.1 changelog entry
* Fix incorrect commands in the release checklist
* Check the crates.io release script in CI, to avoid release failures
* Fix release date in changelog
* Fix workflow typo
* Add --no-confirm to the workflow
* Fix outdated version in release check command
* After the release, add a "remove do-not-merge" step
* Set git identity before committing
* Don't overwrite tweaked versions on upgrade
* cargo release version doesn't support --dependent-version
* Update the same commands in the release checklist
* Commit to git after each change
* And in the checklist
* Add cargo release config
* In CI, allow any branch and provide a fake previous tag name
* Remove redundant `cargo release commit`
* Use a simpler release version process
* Start simplifying the release dry run in CI
* Fix a cargo dependency lint
* just skip the failing crates for now
* Fix the release checklist to match CI
* Use the latest versions of all Zebra dependencies
* Update license description in README for MIT-only crates
* Draft changelog with trivial issues
* Remove trivial issues
* Update changelog entries as of commit 2a31972 and PR #7103
* Update mainnet and testnet checkpoints as of 2023-06-30
* chore: Release
* Estimate release height for Zebra v1.0.1
Block height 2139118 at 2023-06-30 01:55:38 UTC
Release is likely to be 2023-07-01
2139118 + 1152 * 3 = 2142574
Then round up to the nearest 1000.
* Add methods for loading entire column families from the database
* Add a method that loads all the sprout trees from the database
* Add snapshot tests for sprout note commitment trees
* Add round-trip proptests for tree root database serialization
* Add a manual sprout note commitment tree database serialization snapshot test
* Add tests for 1,2,4,8 note commitments in a tree
* Remove redundant "rand" package rename in dependencies
* Randomly cache roots rather than only caching even roots
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add and use `FinalizableBlock`
This commit adds `FinalizableBlock`, and uses it instead of
`ContextuallyVerifiedBlockWithTrees` in `commit_finalized_direct()`
* Use `ContextuallyVerifiedBlockWithTrees`
This commit passes `ContextuallyVerifiedBlockWithTrees` instead of
passing separate `finalized`, `history_tree` and `note_commitment_trees`
when storing blocks in the finalized state.
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* add docs to new methods
* fix existing doc
* rename `ContextuallyVerifiedBlockWithTrees` to `SemanticallyVerifiedBlockWithTrees`
* Refactor docs
* Refactor comments
* Add missing docs, fix typo
* Fix rustfmt
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Remove openssl dependency and prevent it coming back
* Put the arguments in the right place
* Put comment in the right place
* Add a default-docker feature to zebrad and use it in workflows and Docker files
* Fix a comment typo
* Make sure that Docker production builds don't use openssl
* Rename feature to default-release-binaries
* Use Arc::into_inner() to avoid potential concurrency issues
* Remove some outdated clippy lint workarounds (fixed in Rust 1.66)
* Update the required Rust version to 1.70
* Refactor `CheckpointVerifiedBlock`
This commit turns `CheckpointVerifiedBlock` into a wrapper of
`SemanticallyVerifiedBlock` since both structs have the same fields.
* Refactor `ContextuallyVerifiedBlockWithTrees`
This commit uses `SemanticallyVerifiedBlock` in
`ContextuallyVerifiedBlockWithTrees` instead of
`CheckpointVerifiedBlock`.
* fix: prevent progress bar from panicking using workaround
* remove more progress bar lengths
* re-add set_len() calls commented out; restore one that shouldn't cause panics
* Update zebra-state/src/service/non_finalized_state.rs
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: teor <teor@riseup.net>
* change(crates): Publish tower-batch-control
* Cargo fields for zebra-test
* Cargo fields for tower-fallback
* Cargo fields for zebra-chain
* Cargo fields for zebra-consensus
* Cargo fields for zebra-network
* Cargo fields for zebra-node-services
* Cargo fields for zebra-rpc
* Cargo fields for zebra-script
* Cargo fields for zebra-state
* Cargo fields for zebrad
* Cargo fields for zebra-test
* Include zebra-test version and path for zebra-chain
* Include zebra-chain version and path for zebra-node-services
* Include zebra-chain version and path for zebra-script
* Include zebra-chain and zebra-test versions and paths for zebra-state
* Include zebra-* versions and paths for zebra-consensus
* Include zebra-* and tower-fallback versions and paths for tower-batch-control
* Include tower-* versions and paths for zebra-consensus
* Fixed tower-fallback license string
* Remove versions for dev-dependencise
* Update zebra-rpc/Cargo.toml
Co-authored-by: teor <teor@riseup.net>
* tower-batch-control license MUST remain MIT only
* Align zebra-network with upstream tower's MIT license
* Update zebra-test/Cargo.toml
Co-authored-by: teor <teor@riseup.net>
* Update zebra-utils/Cargo.toml
Co-authored-by: teor <teor@riseup.net>
* fix(legal): Use correct licenses for tower-batch-control and zebra-network (#6927)
* Don't modify whitespace
* Resolve rustdoc and cargo-release warnings
* Add missing zebra internal dependency versions
* Workaround missing zebra-utils test dependency in zebrad test-only feature
* Fix missing zebra internal dependency versions
* Add README, home page, and search terms to every crate
* Fix README publish warnings
* Change tower crates to beta versions
* Add zebra-utils to the list of allowed unused dependencies
* Update tower-batch-control/Cargo.toml
* Update tower-fallback/Cargo.toml
---------
Co-authored-by: teor <teor@riseup.net>
teor
dependabot[bot]
Marek
Alfredo Garcia
Arya
Conrado Gouvea
Deirdre Connolly