* Upgrade the scanner database major version to 1
* Update format docs
* Change the high-level scanner db format
* Change the scanner serialization formats
* Fix value format and tests
* Fix incorrect types
* Update documentation
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Update multiple key docs
* Add a TODO for the tree size
* Fix scan() docs
* Split out a scan_height...() function
* Clarify it's the state database we're using for scanning heights
Co-authored-by: Marek <mail@marek.onl>
* Remove unused import
---------
Co-authored-by: Marek <mail@marek.onl>
* Derive & impl helper traits from `std`
* Create `compact_to_v4` fn
* Create `fake_block` fn
* Refactor existing tests to use the new functions
* Cosmetics
* Refactor docs
* Put `Default` behind `cfg_attr(test)`
Rationale
---------
We avoid implementing `Default` on consensus-critical types because it's
easy to miss an incorrect use in a review. It's easy to hide a
`default()` in a call like `unwrap_or_default()` or even more subtle
methods.
---------
Co-authored-by: teor <teor@riseup.net>
* Fix availability of tokio::time in scanner
* Create a function that parses a key into a list of keys
* Pass a ChainTipChange to the scanner function
* Convert a scanned block to a sapling result
* Make it easier to pass keys and blocks
* Increase scanner wait times
* Parse keys once at the start of the scan
* Get a block from the state instead of the tip
* Don't log secret keys, only log every 100,000 blocks
* Scan each block and add the results to storage
* Move blocking tasks into spawn_blocking()
* Update the acceptance test
* Use a dummy sapling tree size
* Use a larger dummy size
* Expose IntoDisk and FromDisk in zebra-state
* Implement database serialization for SaplingScanningKey Strings
* Implement serialization for Vec<SaplingScannedResult> (Vec<transaction::Hash>)
* Implement seralization for SaplingScannedDatabaseIndex
* Add an is_empty() method
* Add a read method for a specific index, and document it
* Implement writing scanner results to the database
* Make read name more explicit
* Implement writing scanner keys
* Implement reading sapling keys
* Spawn blocking tasks correctly in async code
* Change storage results methods to use the database
* Update tests that use storage
* Use spawn_blocking() for database methods
* Change the check interval to slightly less than the block interval
* Expose raw database methods with shielded-scan
* fix `scan_task_starts` test
* minor doc change in test
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Create an empty storage/db module
* Use ephemeral storage in tests
* Populate storage inside new() method
* Move scanner setup into an init() method
* Pass the network to scanner init
* Create a database but don't actually use it
* Skip shutdown format checks when skipping format upgrades
* Allow the scanner to skip launching format upgrades in production
* Refactor skipping format upgrades so it is consistent
* Allow checking configs for equality
* Restore Network import
* Run clippy and build all targets on all crates individually
* Fix prod and test features for scanner deps
* Standardise dependency order
* Remove unnecessary async in tests
* Fix an unused import in a test
* Work around a no space left on device error
* Actually just use a larger runner
* Wrap `zcash_client_backend::scanning::scan_block`
* Use the new `scan_block` fn
* Use full path for `zcash::primitives::Network`
* Add docs for `scan_block`
* Impl `From` for networks in `zcash_primitives`
* Update zebra-scan/src/tests.rs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Move code from `tests.rs` to `scan.rs`
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Provide access to DiskDb and DiskWriteBatch outside the state using a feature
* Actually let's export ZebraDb for the format upgrade code
* Pass column families to ZebraDb as an argument
* Allow the database kind to be changed in config.rs
* Use the state kind in finalized_state.rs
* Allow different database kinds in ZebraDb, but don't move the upgrade code yet
* Allow different database kinds in DiskDb
* Allow different database kinds in upgrade.rs, but don't split the upgrade code out yet
* Add new arguments to raw database tests
* Fix doc links
* Fix internal imports
* Fix unused code
* Update zebrad version metadata
* Create a specific state database delete function
* Fix state exports
* Fix zebrad tests
* Fix zebrad state write tests
* Make CI run again
* Fix dead code warnings for test methods
* Remove unnecessary async on some tests
* Fix logging required by tests
* Fix logging required in test itself
* Fix variable names
* Try to copy the message and add regexes
* Update Connection debug impl missed in previous PRs
* Add an initial_cached_addrs argument to Connection::new()
* Stop sending version and remote IP peers directly to the address book
* Remove the NeverAttemptedAlternate peer state
* Remove the NewAlternate MetaAddrChange
* Remove the never_attempted_alternate AddressBookMetrics
* Update CandidateSet docs and add a Debug impl
* connect zebrad with zebra-scan
* remove unwrap
* use tokio::sleep
* fix the task handler
* Don't panic on an empty state
---------
Co-authored-by: teor <teor@riseup.net>
* Update Connection debug impl missed in previous PRs
* Add an initial_cached_addrs argument to Connection::new()
* Stop sending version and remote IP peers directly to the address book
* Update zebra-network/src/peer/connection.rs
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: Marek <mail@marek.onl>
* Avoid a race condition in the 25.3.0 state upgrade
* Look for the most recent sprout tree height if needed
* Get the latest history tree not the tip height history tree
* Discard keys provided by low level database method
* Remove extra ;
* Provide key types & rustfmt
* Fix weird closure type syntax
* Update zebra-state/src/service/finalized_state/disk_format/upgrade/fix_tree_key_type.rs
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* Skip external branches in ci-unit-tests-docker.yml
* Patch external repositories in ci-unit-tests-docker.patch.yml
* Revert ci-unit-tests-docker.patch.yml
* Add a separate external PR patch in ci-unit-tests-docker.patch-external.yml
* Document ci-unit-tests-docker.patch.yml
* Document ci-unit-tests-docker.yml
* Skip dependabot config validation for external PRs
* Trivial Rust code modification to make Rust workflows run
* Use correct method for checking source branch refs
* Fix the check condition
* Simplify job structure
* Use correct skip condition
* Make the start docs visible so we can link to them
* Skip integration tests on external PRs
* Allow external PRs to pass CD Docker tests
* Allow external PRs to pass Docs build and deploy
* Updates ADDR_RESPONSE_LIMIT_DENOMINATOR to 4
* Moves logic getting a fraction of Zebra's peers to a method in the address book
* Adds and uses CachedPeerAddrs struct in inbound service
* moves and documents constant
* fixes test
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* updates docs
* renames sanitized_window method
* renames CachedPeerAddrs to CachedPeerAddrResponse
* updates test
* moves try_refresh to per request
* Make unused sanitization method pub(crate)
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* moves CachedPeerAddrResponse to a module
* updates unit test
* fixes unit test
* removes unnecessary condition
* clears cached getaddr response if it can't refresh for over a minute after the refresh time
* tests that inbound service gives out the same addresses for every Peers request before the refresh interval
* Applies suggestion from code review
* fixes doc link
* renames constant
* Fix docs on new constant
* applies suggestion from code review
* uses longer cache expiry time
* Adds code comments
---------
Co-authored-by: teor <teor@riseup.net>
* Reformat a method to make related code clearer
* Limit how many addresses we take from each peer response
* Only put extra addresses back in the cache if there are actually any extras
* Refactor address caching into a single method
* Use the same method for updating the peer cache and taking peers from it
* Choose response peers at random, and partially randomly drop excess peers
* Update docs for this fix
* Fix an index bug and clarify a comment
* Add a proptest for cache updates
* Refactor return type of poll_discover()
* Simplify poll_ready() by removing preselected peers
* Fix peer set readiness check
* Pass task context correctly to background tasks
* Make poll_discover() return Pending
* Make poll_inventory() return Pending
* Make poll_unready() return Poll::Pending
* Simplify with futures::ready!() and ?
* When there are no peers, wake on newly ready peers, or new peers, in that order
* Preserve the original waker when there are no unready peers
* Fix polling docs and remove unnecessary code
* Make sure we're ignoring Poll::Pending not Result
* Make panic checking method names clearer
* Fix connection Client task wakeups and error handling
* Cleanup connection panic handling and add wakeup docs
* Fix connection client task wakeups to prevent hangs
* Simplify error and pending handling
* Clarify inventory set behaviour
* Define peer set poll_* methods so they return Ok if they do something
* Clarify documentation
Co-authored-by: Arya <aryasolhi@gmail.com>
* Fix test that depended on preselected peers
* Check ready peers for errors before sending requests to them
* Fix a hanging test by not waiting for irrelevant actions
* Only remove cancel handles when they are required
* fix incorrect panic on termination setting
* Clarify method comments
Co-authored-by: Arya <aryasolhi@gmail.com>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* add a basic RAM database for the scanner
* specify a crate version for zebra-chain dependency
* add a type for sapling keys
* rename everything to reflect sapling
* change some storage methods
* get started with the blockchain scanner poc
* rustfmt
* fix the tests
* Reads blocks from db
* Adds conversion functions
* scans blocks and counts transactions
* fix bug
* split into 2 tests
* add duplicated dependencies to deny.toml
* upgrade zebra-scanner version
* try removing ecc duplicated dependencies
* try fix deny.toml
* remove unintended paste from deny.toml
* remove duplicated code from the other test
* remove strict version of `zcash_primitives` crate
* change description
* remove feture
* remove tokio features
* change lib doc
* add more documentation
* change expect
* do not use default in compact block creation
* more docs
* add more checks to test
* remove zebra-consensus dependency
* move all deps to dev-deps
* change crate version
* rename crate to zebra-scan
* lock file
* add test for zecpages populated state
* scans all cached blocks for zecpages viewing key expecting Ok results.
* use test blocks
* fixes test
* fix expect messages
* Discard changes to Cargo.lock
* Discard changes to deny.toml
---------
Co-authored-by: arya2 <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Stop using `SemanticallyVerifiedBlockWithTrees`
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Doc that default treestate corresponds to genesis
* Doc the diff between default values of tree roots
---------
Co-authored-by: teor <teor@riseup.net>
teor
dependabot[bot]
Marek
Alfredo Garcia
Arya