Commit Graph

10 Commits

Author SHA1 Message Date
teor 4306a00f3c
Scan blocks with sapling keys and write the results to the database (#8040)
* Fix availability of tokio::time in scanner

* Create a function that parses a key into a list of keys

* Pass a ChainTipChange to the scanner function

* Convert a scanned block to a sapling result

* Make it easier to pass keys and blocks

* Increase scanner wait times

* Parse keys once at the start of the scan

* Get a block from the state instead of the tip

* Don't log secret keys, only log every 100,000 blocks

* Scan each block and add the results to storage

* Move blocking tasks into spawn_blocking()

* Update the acceptance test

* Use a dummy sapling tree size

* Use a larger dummy size
2023-12-03 21:58:48 +00:00
teor 8c717c92dd
change(scan): Create a scanner storage database, but don't use it yet (#8031)
* Create an empty storage/db module

* Use ephemeral storage in tests

* Populate storage inside new() method

* Move scanner setup into an init() method

* Pass the network to scanner init

* Create a database but don't actually use it

* Skip shutdown format checks when skipping format upgrades

* Allow the scanner to skip launching format upgrades in production

* Refactor skipping format upgrades so it is consistent

* Allow checking configs for equality

* Restore Network import
2023-11-30 12:59:15 +00:00
teor cb9452c5e3
change(ci): When building crates individually, build all targets, and run clippy (#8024)
* Run clippy and build all targets on all crates individually

* Fix prod and test features for scanner deps

* Standardise dependency order

* Remove unnecessary async in tests

* Fix an unused import in a test

* Work around a no space left on device error

* Actually just use a larger runner
2023-11-30 00:51:20 +00:00
Marek e00a762856
Create a `scan_block` function to use across scanning tasks (#7994)
* Wrap `zcash_client_backend::scanning::scan_block`

* Use the new `scan_block` fn

* Use full path for `zcash::primitives::Network`

* Add docs for `scan_block`

* Impl `From` for networks in `zcash_primitives`

* Update zebra-scan/src/tests.rs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Move code from `tests.rs` to `scan.rs`

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2023-11-28 21:58:18 +00:00
dependabot[bot] 35a7764b01
build(deps): bump the prod group with 3 updates (#8017)
Bumps the prod group with 3 updates: [clap](https://github.com/clap-rs/clap), [serde](https://github.com/serde-rs/serde) and [inferno](https://github.com/jonhoo/inferno).


Updates `clap` from 4.4.8 to 4.4.9
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.4.8...v4.4.9)

Updates `serde` from 1.0.192 to 1.0.193
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.192...v1.0.193)

Updates `inferno` from 0.11.18 to 0.11.19
- [Changelog](https://github.com/jonhoo/inferno/blob/main/CHANGELOG.md)
- [Commits](https://github.com/jonhoo/inferno/compare/v0.11.18...v0.11.19)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: inferno
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-28 03:29:47 +00:00
Alfredo Garcia 0f24c311a4
feat(zebra-scan): Connect with zebrad (#7989)
* connect zebrad with zebra-scan

* remove unwrap

* use tokio::sleep

* fix the task handler

* Don't panic on an empty state

---------

Co-authored-by: teor <teor@riseup.net>
2023-11-27 12:52:48 +00:00
Alfredo Garcia a22c8d5f42
feat(config): Allow to add keys to be scanned by the zebra-scan crate to config (#7949)
* allow user to add sapling keys to config

* apply code review suggestions

Co-authored-by: teor <teor@riseup.net>

---------

Co-authored-by: teor <teor@riseup.net>
2023-11-16 01:32:22 +00:00
Alfredo Garcia 732ee01443
feat(scanner): Add a very basic RAM database to store keys and scan results (#7942)
* add a basic RAM database for the scanner

* specify a crate version for zebra-chain dependency

* add a type for sapling keys

* rename everything to reflect sapling

* change some storage methods
2023-11-15 23:06:23 +00:00
dependabot[bot] 461c5aa680
build(deps): bump the prod group with 7 updates (#7938)
Bumps the prod group with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [clap](https://github.com/clap-rs/clap) | `4.4.7` | `4.4.8` |
| [serde](https://github.com/serde-rs/serde) | `1.0.190` | `1.0.192` |
| [toml](https://github.com/toml-rs/toml) | `0.8.6` | `0.8.8` |
| [tokio](https://github.com/tokio-rs/tokio) | `1.33.0` | `1.34.0` |
| [sentry](https://github.com/getsentry/sentry-rust) | `0.31.7` | `0.31.8` |
| [inferno](https://github.com/jonhoo/inferno) | `0.11.17` | `0.11.18` |
| [proptest](https://github.com/proptest-rs/proptest) | `1.3.1` | `1.4.0` |


Updates `clap` from 4.4.7 to 4.4.8
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.4.7...v4.4.8)

Updates `serde` from 1.0.190 to 1.0.192
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.190...v1.0.192)

Updates `toml` from 0.8.6 to 0.8.8
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.8.6...toml-v0.8.8)

Updates `tokio` from 1.33.0 to 1.34.0
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.33.0...tokio-1.34.0)

Updates `sentry` from 0.31.7 to 0.31.8
- [Release notes](https://github.com/getsentry/sentry-rust/releases)
- [Changelog](https://github.com/getsentry/sentry-rust/blob/master/CHANGELOG.md)
- [Commits](https://github.com/getsentry/sentry-rust/compare/0.31.7...0.31.8)

Updates `inferno` from 0.11.17 to 0.11.18
- [Changelog](https://github.com/jonhoo/inferno/blob/main/CHANGELOG.md)
- [Commits](https://github.com/jonhoo/inferno/compare/v0.11.17...v0.11.18)

Updates `proptest` from 1.3.1 to 1.4.0
- [Release notes](https://github.com/proptest-rs/proptest/releases)
- [Changelog](https://github.com/proptest-rs/proptest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/proptest-rs/proptest/compare/v1.3.1...v1.4.0)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: toml
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: sentry
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: inferno
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: proptest
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-15 01:03:57 +00:00
Alfredo Garcia 86e468f3b0
poc(scanner): get started with the blockchain scanner proof of concept (#7758)
* get started with the blockchain scanner poc

* rustfmt

* fix the tests

* Reads blocks from db

* Adds conversion functions

* scans blocks and counts transactions

* fix bug

* split into 2 tests

* add duplicated dependencies to deny.toml

* upgrade zebra-scanner version

* try removing ecc duplicated dependencies

* try fix deny.toml

* remove unintended paste from deny.toml

* remove duplicated code from the other test

* remove strict version of `zcash_primitives` crate

* change description

* remove feture

* remove tokio features

* change lib doc

* add more documentation

* change expect

* do not use default in compact block creation

* more docs

* add more checks to test

* remove zebra-consensus dependency

* move all deps to dev-deps

* change crate version

* rename crate to zebra-scan

* lock file

* ifix cargo.lock

* remove internal dev dependencies versions

Co-authored-by: teor <teor@riseup.net>

* fix docs url

Co-authored-by: teor <teor@riseup.net>

* fix expect messages

Co-authored-by: teor <teor@riseup.net>

* remove duplicated in deny.toml

Co-authored-by: teor <teor@riseup.net>

* add a comment about moving code to production

---------

Co-authored-by: arya2 <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2023-11-08 22:05:51 +00:00