Commit Graph

9 Commits

Author SHA1 Message Date
teor 4306a00f3c
Scan blocks with sapling keys and write the results to the database (#8040)
* Fix availability of tokio::time in scanner

* Create a function that parses a key into a list of keys

* Pass a ChainTipChange to the scanner function

* Convert a scanned block to a sapling result

* Make it easier to pass keys and blocks

* Increase scanner wait times

* Parse keys once at the start of the scan

* Get a block from the state instead of the tip

* Don't log secret keys, only log every 100,000 blocks

* Scan each block and add the results to storage

* Move blocking tasks into spawn_blocking()

* Update the acceptance test

* Use a dummy sapling tree size

* Use a larger dummy size
2023-12-03 21:58:48 +00:00
teor db05845f98
change(scan): Use the on-disk database for keys and results (#8036)
* Expose IntoDisk and FromDisk in zebra-state

* Implement database serialization for SaplingScanningKey Strings

* Implement serialization for Vec<SaplingScannedResult> (Vec<transaction::Hash>)

* Implement seralization for SaplingScannedDatabaseIndex

* Add an is_empty() method

* Add a read method for a specific index, and document it

* Implement writing scanner results to the database

* Make read name more explicit

* Implement writing scanner keys

* Implement reading sapling keys

* Spawn blocking tasks correctly in async code

* Change storage results methods to use the database

* Update tests that use storage

* Use spawn_blocking() for database methods

* Change the check interval to slightly less than the block interval

* Expose raw database methods with shielded-scan

* fix `scan_task_starts` test

* minor doc change in test

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2023-11-30 21:27:46 +00:00
teor 8c717c92dd
change(scan): Create a scanner storage database, but don't use it yet (#8031)
* Create an empty storage/db module

* Use ephemeral storage in tests

* Populate storage inside new() method

* Move scanner setup into an init() method

* Pass the network to scanner init

* Create a database but don't actually use it

* Skip shutdown format checks when skipping format upgrades

* Allow the scanner to skip launching format upgrades in production

* Refactor skipping format upgrades so it is consistent

* Allow checking configs for equality

* Restore Network import
2023-11-30 12:59:15 +00:00
Alfredo Garcia 1708f9d946
change(zebra-scan): Scan only one key per backend call (#8034)
* scan one key per backend call

* fix docs
2023-11-30 00:51:32 +00:00
teor cb9452c5e3
change(ci): When building crates individually, build all targets, and run clippy (#8024)
* Run clippy and build all targets on all crates individually

* Fix prod and test features for scanner deps

* Standardise dependency order

* Remove unnecessary async in tests

* Fix an unused import in a test

* Work around a no space left on device error

* Actually just use a larger runner
2023-11-30 00:51:20 +00:00
Marek e00a762856
Create a `scan_block` function to use across scanning tasks (#7994)
* Wrap `zcash_client_backend::scanning::scan_block`

* Use the new `scan_block` fn

* Use full path for `zcash::primitives::Network`

* Add docs for `scan_block`

* Impl `From` for networks in `zcash_primitives`

* Update zebra-scan/src/tests.rs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Move code from `tests.rs` to `scan.rs`

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2023-11-28 21:58:18 +00:00
Alfredo Garcia 732ee01443
feat(scanner): Add a very basic RAM database to store keys and scan results (#7942)
* add a basic RAM database for the scanner

* specify a crate version for zebra-chain dependency

* add a type for sapling keys

* rename everything to reflect sapling

* change some storage methods
2023-11-15 23:06:23 +00:00
Alfredo Garcia e5e89ec549
poc(scanner): add a populated state test for ZECpages viewing key (#7916)
* get started with the blockchain scanner poc

* rustfmt

* fix the tests

* Reads blocks from db

* Adds conversion functions

* scans blocks and counts transactions

* fix bug

* split into 2 tests

* add duplicated dependencies to deny.toml

* upgrade zebra-scanner version

* try removing ecc duplicated dependencies

* try fix deny.toml

* remove unintended paste from deny.toml

* remove duplicated code from the other test

* remove strict version of `zcash_primitives` crate

* change description

* remove feture

* remove tokio features

* change lib doc

* add more documentation

* change expect

* do not use default in compact block creation

* more docs

* add more checks to test

* remove zebra-consensus dependency

* move all deps to dev-deps

* change crate version

* rename crate to zebra-scan

* lock file

* add test for zecpages populated state

* scans all cached blocks for zecpages viewing key expecting Ok results.

* use test blocks

* fixes test

* fix expect messages

* Discard changes to Cargo.lock

* Discard changes to deny.toml

---------

Co-authored-by: arya2 <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2023-11-08 23:46:47 +00:00
Alfredo Garcia 86e468f3b0
poc(scanner): get started with the blockchain scanner proof of concept (#7758)
* get started with the blockchain scanner poc

* rustfmt

* fix the tests

* Reads blocks from db

* Adds conversion functions

* scans blocks and counts transactions

* fix bug

* split into 2 tests

* add duplicated dependencies to deny.toml

* upgrade zebra-scanner version

* try removing ecc duplicated dependencies

* try fix deny.toml

* remove unintended paste from deny.toml

* remove duplicated code from the other test

* remove strict version of `zcash_primitives` crate

* change description

* remove feture

* remove tokio features

* change lib doc

* add more documentation

* change expect

* do not use default in compact block creation

* more docs

* add more checks to test

* remove zebra-consensus dependency

* move all deps to dev-deps

* change crate version

* rename crate to zebra-scan

* lock file

* ifix cargo.lock

* remove internal dev dependencies versions

Co-authored-by: teor <teor@riseup.net>

* fix docs url

Co-authored-by: teor <teor@riseup.net>

* fix expect messages

Co-authored-by: teor <teor@riseup.net>

* remove duplicated in deny.toml

Co-authored-by: teor <teor@riseup.net>

* add a comment about moving code to production

---------

Co-authored-by: arya2 <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2023-11-08 22:05:51 +00:00