Commit Graph

150 Commits

Author SHA1 Message Date
Gustavo Valverde aec07f24ff
ref(ci): consolidate cached states workflows and scripts (#8865)
* ref(ci): consolidate cached states workflows and scripts

We've been using multiple approaches to locate and retrieve cached states in GCP. However, this has made it difficult to reuse the same methods across new workflows or different scenarios.

To address this, we've streamlined the process to make it more reusable in other contexts. This change will support deploying instances from both the `main` branch and `release`, simplifying future implementations and speeding up the process.

Changes:
- Use a single bash script (`gcp-get-cached-disks.sh`) to get cached states names and availability
- Move script logic from `sub-find-cached-disks.yml` to `gcp-get-cached-disks.sh` and adapt `sub-find-cached-disks.yml` to allow to output available disks and disks names.
- Simplify parameters usage in `sub-deploy-integration-tests-gcp.yml` and convert the `Find ${{ inputs.test_id }} cached state disk` step into an independent job, to be able to use the `sub-find-cached-disks.yml` reusable workflow
- Remove repetition in `sub-ci-integration-tests-gcp.yml`

* ref(tests): Use the `ZEBRA_CACHED_STATE_DIR` env var across tests

We had a technical debt with some tests using a hardcoded value for the cache directory (`/zebrad-cache`), which generated inconsistency across disks and cached states directories.

Changes:
- Allow sync tests to use the `ZEBRA_CACHED_STATE_DIR` as the cache directory, if specified
- Update the `entrypoint.sh` to reflect this change
- Add the `ZEBRA_CACHED_STATE_DIR` variable to the missing tests in `sub-ci-integration-tests-gcp.yml`, and remove extra parameters to call reusable workflows.
2024-09-19 12:31:33 +00:00
Alfredo Garcia 1511997829
fix clippy lints (#8855) 2024-09-06 15:24:15 +00:00
Arya 9fbcb4ccd2
Updates ignored failure messages in lightwalletd_integration test (#8784) 2024-08-20 22:13:29 +00:00
Arya 988dd55e48
change(network-params): Configurable Testnet funding streams (#8718)
* Addresses clippy lints

* checks network magic and returns early from `is_regtest()`

* Moves  `subsidy.rs` to `zebra-chain`, refactors funding streams into structs, splits them into pre/post NU6 funding streams, and adds them as a field on `testnet::Parameters`

* Replaces Vec with HashMap, adds `ConfiguredFundingStreams` type and conversion logic with constraints.

Minor refactors

* Empties recipients list

* Adds a comment on num_addresses calculation being invalid for configured Testnets, but that being okay since configured testnet parameters are checked when they're being built

* Documentation fixes, minor cleanup, renames a test, adds TODOs, and fixes test logic

* Removes unnecessary `ParameterSubsidy` impl for &Network, adds docs and TODOs

---------

Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
2024-07-29 16:15:09 +00:00
Arya 6181ebe239
Addresses clippy lints (#8723) 2024-07-26 14:29:35 +00:00
dependabot[bot] 0fdd26431e
build(deps): bump the prod group across 1 directory with 15 updates (#8692)
* build(deps): bump the prod group across 1 directory with 15 updates

Bumps the prod group with 15 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [clap](https://github.com/clap-rs/clap) | `4.5.8` | `4.5.9` |
| [tokio](https://github.com/tokio-rs/tokio) | `1.38.0` | `1.38.1` |
| [tinyvec](https://github.com/Lokathor/tinyvec) | `1.7.0` | `1.8.0` |
| [thiserror](https://github.com/dtolnay/thiserror) | `1.0.61` | `1.0.62` |
| [inferno](https://github.com/jonhoo/inferno) | `0.11.19` | `0.11.20` |
| [hyper](https://github.com/hyperium/hyper) | `0.14.29` | `1.3.1` |
| [metrics-exporter-prometheus](https://github.com/metrics-rs/metrics) | `0.15.1` | `0.15.3` |
| [prost](https://github.com/tokio-rs/prost) | `0.12.6` | `0.13.1` |
| [tonic](https://github.com/hyperium/tonic) | `0.11.0` | `0.12.0` |
| [vergen](https://github.com/rustyhorde/vergen) | `8.3.1` | `8.3.2` |
| [tonic-build](https://github.com/hyperium/tonic) | `0.11.0` | `0.12.0` |
| [serde_with](https://github.com/jonasbb/serde_with) | `3.8.3` | `3.9.0` |
| [bytes](https://github.com/tokio-rs/bytes) | `1.6.0` | `1.6.1` |
| [tonic-reflection](https://github.com/hyperium/tonic) | `0.11.0` | `0.12.0` |
| [syn](https://github.com/dtolnay/syn) | `2.0.69` | `2.0.71` |



Updates `clap` from 4.5.8 to 4.5.9
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/clap_complete-v4.5.8...v4.5.9)

Updates `tokio` from 1.38.0 to 1.38.1
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.38.0...tokio-1.38.1)

Updates `tinyvec` from 1.7.0 to 1.8.0
- [Changelog](https://github.com/Lokathor/tinyvec/blob/main/CHANGELOG.md)
- [Commits](https://github.com/Lokathor/tinyvec/compare/v1.7.0...v1.8.0)

Updates `thiserror` from 1.0.61 to 1.0.62
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.61...1.0.62)

Updates `inferno` from 0.11.19 to 0.11.20
- [Changelog](https://github.com/jonhoo/inferno/blob/main/CHANGELOG.md)
- [Commits](https://github.com/jonhoo/inferno/compare/v0.11.19...v0.11.20)

Updates `hyper` from 0.14.29 to 1.3.1
- [Release notes](https://github.com/hyperium/hyper/releases)
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.29...v1.3.1)

Updates `metrics-exporter-prometheus` from 0.15.1 to 0.15.3
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/compare/metrics-exporter-prometheus-v0.15.1...metrics-exporter-prometheus-v0.15.3)

Updates `prost` from 0.12.6 to 0.13.1
- [Release notes](https://github.com/tokio-rs/prost/releases)
- [Changelog](https://github.com/tokio-rs/prost/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/prost/compare/v0.12.6...v0.13.1)

Updates `tonic` from 0.11.0 to 0.12.0
- [Release notes](https://github.com/hyperium/tonic/releases)
- [Changelog](https://github.com/hyperium/tonic/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/tonic/compare/v0.11.0...v0.12.0)

Updates `vergen` from 8.3.1 to 8.3.2
- [Release notes](https://github.com/rustyhorde/vergen/releases)
- [Commits](https://github.com/rustyhorde/vergen/commits)

Updates `tonic-build` from 0.11.0 to 0.12.0
- [Release notes](https://github.com/hyperium/tonic/releases)
- [Changelog](https://github.com/hyperium/tonic/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/tonic/compare/v0.11.0...v0.12.0)

Updates `serde_with` from 3.8.3 to 3.9.0
- [Release notes](https://github.com/jonasbb/serde_with/releases)
- [Commits](https://github.com/jonasbb/serde_with/compare/v3.8.3...v3.9.0)

Updates `bytes` from 1.6.0 to 1.6.1
- [Release notes](https://github.com/tokio-rs/bytes/releases)
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.6.0...v1.6.1)

Updates `tonic-reflection` from 0.11.0 to 0.12.0
- [Release notes](https://github.com/hyperium/tonic/releases)
- [Changelog](https://github.com/hyperium/tonic/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/tonic/compare/v0.11.0...v0.12.0)

Updates `syn` from 2.0.69 to 2.0.71
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/2.0.69...2.0.71)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: tinyvec
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: inferno
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: hyper
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod
- dependency-name: metrics-exporter-prometheus
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: prost
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: tonic
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: vergen
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: tonic-build
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: serde_with
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: bytes
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: tonic-reflection
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: syn
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>

* make changes for hyper 1.0

* unrelated clippy

* fix panic

* update denies

* add missing deny.toml

* add more denies for all features

* add timeout feature to tower in zebra-grpc

* cargo vet

* cargo vet prune

* codespell

* change waiting message

* remove hyper direct dependency from zebra-rpc

* fix doc typos

* simplify tracing endpoint

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2024-07-23 15:44:31 +00:00
Arya 42b298b074
add(network-params): Allow configuring NU6 activation height on Regtest (#8700)
* Allows configuration for regtest nu6 activation height

* Adds test config
2024-07-22 11:36:35 +00:00
Alfredo Garcia 1238ec0c63
tests(scanner): Move zebra scanner tests to binary (#8659)
* readd the scanner tests in the new binary context

* remove commented out tests from zebrad

* add Cargo.lock

* add local copy of `ZECPAGES_SAPLING_VIEWING_KEY` to binary scanning tests

* moves scanner-results-reader to zebra-scan, adds zebrad bin to zebra-scan, adds a short sleep before trying to connect to gRPC server in test

* always include zebrad dependency in zebra-scan

* renames zebrad bin in zebra-scan to zebrad-for-scanner

* skip test when a cached state is missing

* remove scanning config from zebrad

* move `scan_task_commands` out of zebrad acceptance tests

* remove scanner config from zebrad

* do not panic if env var is not there

* re enable start_scan_where left, remove shielded-scan feature foir scanning tests in CI

* change app names

* uncomment test

* fix the scanner tests in CI by changing the test command

* add log needed by CI?

* move test

* rename trait, docs fix

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2024-07-15 21:15:04 +00:00
Alfredo Garcia a94b2be0bb
feat(scanner): Add a new `zebra-scanner` binary (#8608)
* add a new `zebra-scanner` binary

* update arguments

* allow birthday in config

* remove required feature

* add `env-filter` feature to `tracing-subscriber` dependency

* use sync task

* codespell

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2024-07-09 18:50:26 +00:00
Arya 4213e82a4f
change(rpc): Adds a TrustedChainSync struct for keeping up with Zebra's non-finalized best chain from a separate process (#8596)
* Adds an init_read_only() fn in zebra-state

* moves elasticsearch initialization to `FinalizedState::new_with_debug()`

* Updates callers of `FinalizedState::{new, new_with_debug}` to pass a bool to try enabling elasticsearch

* Adds a non-finalized read state syncer to zebra-rpc

* moves, removes, updates, or addresses TODOs

* reduces disk IO while waiting for the a new chain tip & updates the chain tip sender when the finalized tip has changed.

* Returns boxed errors from RpcRequestClient methods instead of color_eyre type

* Avoids resetting the non-finalized state when there's an error getting a block unless it has the missing block error code.

* Adds stub for acceptance test(s) and removes outdated TODO

* adds TODOs for testing

* Tests that `ChainTipChange` is updated when the non-finalized best chain grows

* adds a last_chain_tip_hash and uses a FuturesOrdered for getblock requests

* Fixes a pre-flush sync issue by using a secondary db instead of a read-only db

* Moves disk IO to blocking tasks

* Updates acceptance test to how forks are handled

* Checks synced read state for all of the expected blocks

* checks that there isn't a tip change until the best chain changes

* checks for chain tip changes in test

* run test without feature

* fixes lint

* Fixes compilation/test issues

* Adds docs / comments, moves HexData out from behind the getblocktemplate-rpcs feature flag, moves test behind the mining feature flag.

* Fixes lints

* removes syncer and rpc-syncer features

* Fixes test on Windows, applies suggestions from code review

* Updates `POLL_DELAY` documentation

* Updates method docs

* Fixes a test bug

* use rpc-client feature in zebrad production code

* use rpc-client feature in zebra-node-services for building zebra-rpc crate

---------

Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2024-07-09 14:15:47 +00:00
Arya 2419e8a342
read RPC port from logs in `regtest_submit_blocks()` test (#8665) 2024-07-08 23:12:23 +00:00
Arya 979ed0d720
add(book): Describes custom Testnets in the Zebra book (#8636)
* Adds documention to the Zebra book about custom Testnets, allows for configuring custom Testnet genesis hashes, refactors check for compatible custom Testnets, adds a TODO, and uses the default testnet when there's an empty or default `testnet_parameters` field in Zebra's network config.

* Adds example configs to custom Testnet docs, moves note about inbound connections on Regtest to a footnote.

* Adds a changelog entry

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

* Addresses suggestions from code review.

* Apply suggestions from code review

Co-authored-by: Arya <aryasolhi@gmail.com>

* Update book/src/user/custom-testnets.md

---------

Co-authored-by: Marek <mail@marek.onl>
2024-06-28 00:33:17 +00:00
Arya fb424220a8
fix(tests): Address port conflict issues and enable related Windows tests (#8624)
* Uses an OS-assigned port when testing rpc_server_spawn

* reads port from logs and enables 2 acceptance tests on Windows

* generalizes read listen addr from logs fn and re-enables another test

* updates zebra-grpc snapshot tests to use OS-assigned port

* Re-enable grpc vectors tests on Windows while avoiding port conflicts

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2024-06-21 14:32:33 +00:00
Arya c494aa7bd1
change(network): Allow custom testnets to make peer connections and configure more parameters (#8528)
* Allow custom testnets to define more parameters and to make peer connections based on the initial_testnet_peer config field

* Updates latest stored test config

* Update zebra-network/src/config.rs

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Marek <mail@marek.onl>
2024-05-20 13:11:34 +00:00
Arya 5ca40aa58b
add(network): Adds a network magic field to `testnet::Parameters` and the config (#8524)
* Adds a `target_difficulty_limit` field on `testnet::Parameters`

* updates test to increment block nonce until finding a block that's below the difficulty threshold

* increment the nonce while the difficulty is invalid instead of while the difficulty threshold is invalid

* Adds comments

* moves network Magic type from zebra-network to zebra-chain

* Adds `network_magic` field to `testnet::Parameters` and uses the regtest network magic from zcashd

* Add a network magic config field for custom testnets

* Adds/updates tests

* Update zebra-chain/src/parameters/network/testnet.rs

* Adds a link to the Regtest network magic in zcashd

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2024-05-18 20:30:59 +00:00
Arya 99b017e2a4
add(consensus): Add a `target_difficulty_limit` field on `testnet::Parameters` (#8518)
* Adds a `target_difficulty_limit` field on `testnet::Parameters`

* updates test to increment block nonce until finding a block that's below the difficulty threshold

* increment the nonce while the difficulty is invalid instead of while the difficulty threshold is invalid

* Adds comments
2024-05-17 23:06:08 +00:00
Arya 15e1096826
change(consensus): Allow configurable NU5 activation height on Regtest (#8505)
* Allow configurable Nu5 activation height on Regtest

* Fixes test

* removes outdated TODO

* Adds `current_with_activation_height()` method and uses it in `Commitment::from_bytes()`
2024-05-08 14:46:13 +00:00
Alfredo Garcia 239fcc85ba
feat(network): Configurable external address for Version network messages (#8488)
* add configurable external address for Version network messages

* apply suggestions from code review

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2024-05-03 01:07:52 +00:00
Arya 8a786fe6ce
add(consensus): Adds block construction and validation for NU5 from block height 1 for Regtest (#8475)
* Always activate Canopy at Height(1) on Regtest

* Ignores the zip 212 grace period on configured Testnets and Regtest

* - Returns early when there is no Heartwood activation height when creating or updating HistoryTree
- Skips call to `check::legacy_chain()` when no NU5 activation height is set (it would return immediately anyway)
- Replaces `.map()` with `.filter_map()` in `NetworkUpgrade::target_spacings()`
- Removes outdated TODO

* - When proof of work is disabled, skips checking if Zebra is synced to the network tip in the getblocktemplate method

* Sets full_verifier_utxo_lookahead to Height::MIN instead of panicking

* When network is regtest, skips starting sync task and commits the genesis block if it's missing in the state

* updates/fixes test config

* Adds test for committing Canopy blocks on Regtest

* - Updates median time past and difficulty checks to use fewer than 11/17/28 blocks

* uses SLOW_START_INTERVAL of 0 if PoW is disabled, adds TODOs

* Update getblocktemplate method to return reserved chain history activation root hash, uses Nu5 at height 1

Test passes.

* Updates test to expect NU5 as the default nu activation at Height(1)

* Removes invalid difficulty snapshot

* fixes tests

* removes regtest NU5 activation height config field

* Apply suggestions from code review

* Update zebra-state/src/service/check/difficulty.rs

Co-authored-by: Marek <mail@marek.onl>

* Update zebra-state/src/service/check/difficulty.rs

* Updates docs/comments, renames an argument

---------

Co-authored-by: Marek <mail@marek.onl>
2024-05-02 19:43:42 +00:00
Arya e5d4d739b7
add(consensus/network): Adds `new_regtest()` constructors to `testnet::Parameters` and `Network` (#8413)
* minor cleanup and rename

* Adds an empty NetworkParameters struct to Network::Testnet variant, updates production code.

* Updates tests

* Adds `NetworkKind` and uses it instead of `Network` in `HistoryTreeParts` and `transparent::Address`

* Adds a [network.testnet_parameters] section to the config, uses `NetworkKind` as zebra_network::Config::network field type, and converts 'Network' to `NetworkKind` before serializing

* Applies some suggestions from code review

* Applies suggestions from code review

* returns b58 prefix constants directly to remove From<NetworkKind> impl for zcash_primitives::consensus::Network

* Applies more suggestions from code review.

* moves conversions to zcash_primitives::consensus::Network to where they're used.

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

* rename `network` variables and method names typed as NetworkKind to `network_kind`

* use only test block heights for the network associated with them

* Applies more suggestions from code review.

* Rename `NetworkParameters` to `Parameters` and move it a new `testnet` module

* adds activation heights field

* updates stored test config, adds a quicker test for checking that stored configs can be parsed, adds an intermediate representation of activation heights

* implement Parameters for Network

* Passes &Network directly instead of converting to zp_consensus::Network where there were conversions

* fixes a bad merge (removes a network conversion in zcash_note_encryption)

* Adds a test for the Parameters impl for zebra_chain::Network

* fixes doc links

* - Makes the `activation_heights` config field optional by adding a #[serde(default)]
- Panics if a non-zero activation height is provided for the `Genesis` network upgrade
- Always sets the `Genesis` and `BeforeOverwinter` network upgrade activation heights to 0 and 1, `BeforeOverwinter` could be overwritten by a later network upgrade
- Makes the `activation_heights` field on `Parameters` private, adds/uses an accessor method instead, and adds a builder struct and `build()` method

* small refactor of activation_heights() method

* check that activation heights are in the right order

* Updates `NetworkUpgrade::activation_height()` to return the height of the next NetworkUpgrade if it doesn't find the activation height of `&self`

* checks that the miner address is of TestnetKind on Regtest and update assertion message

* Adds a DNetworkUpgradeActivationHeights struct for better control over how activation heights can be configured

* moves all ordered network upgrades to a constant, adds a no_duplicates test, moves struct with activation heights outside deserialization impl and accepts it in `ParametersBuilder::activation_heights()` instead of a Vec

* panics if any network upgrades are configured to activate at Height(0) because it's reserved for Genesis

* Simplifies the `ParametersBuilder::activation_heights()` method and removes an unnecessary test.

* Adds Sapling HRPs as fields on testnet params. (#8398)

* Applies suggestions from code review.

* Update zebra-chain/src/parameters/network_upgrade.rs

Co-authored-by: Marek <mail@marek.onl>

* Adds `network_name` field and accessor method on `Parameters`, uses it in the `Display` impl for `Network`

* Removes unnecessary `.filter()`

* adds constraints on valid network names and a test

* adds config field for setting network name, adds "with_" prefix to ParameterBuilder setter methods

* Adds `MainnetKind`, `TestnetKind`, and `RegtestKind` to reserved network names

* updates stored test configs and fixes `network_name()` docs

* Adds a `new_regtest()` method on `Network` and `testnet::Parameters`, updates config deserialization to return an error if the initial_testnet_peers include any default initial peers AND contain configured activation heights

* Updates `activates_network_upgrades_correctly` test to check Regtest activation heights (and default Mainnet/Testnet)

* Refactors if-let & match statement into general match statement, removes constraint against including `testnet_parameters` field/section in the config when using `Mainnet` or `Regtest`

* Removes outdated TODO

* Restores `testnet_parameters` section of the latest stored config.

* Adds `with_sapling_hrps()` method and uses it to set the Regtest HRPs in `new_regtest()`.

Adds a test for Sapling HRP validation

* Checks that default Mainnet/Testnet/Regtest Sapling HRPs pass validation in `with_sapling_hrps()`

* Uses the correct constant in test

* Adds `is_regtest()` methods

* Updates test docs

* drop custom panic hooks after expected panics

---------

Co-authored-by: Marek <mail@marek.onl>
2024-04-25 04:04:05 +00:00
Arya 275e99ec72
add(chain): Adds a `network_name` field to `testnet::Parameters` (#8411)
* minor cleanup and rename

* Adds an empty NetworkParameters struct to Network::Testnet variant, updates production code.

* Updates tests

* Adds `NetworkKind` and uses it instead of `Network` in `HistoryTreeParts` and `transparent::Address`

* Adds a [network.testnet_parameters] section to the config, uses `NetworkKind` as zebra_network::Config::network field type, and converts 'Network' to `NetworkKind` before serializing

* Applies some suggestions from code review

* Applies suggestions from code review

* returns b58 prefix constants directly to remove From<NetworkKind> impl for zcash_primitives::consensus::Network

* Applies more suggestions from code review.

* moves conversions to zcash_primitives::consensus::Network to where they're used.

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

* rename `network` variables and method names typed as NetworkKind to `network_kind`

* use only test block heights for the network associated with them

* Applies more suggestions from code review.

* Rename `NetworkParameters` to `Parameters` and move it a new `testnet` module

* adds activation heights field

* updates stored test config, adds a quicker test for checking that stored configs can be parsed, adds an intermediate representation of activation heights

* implement Parameters for Network

* Passes &Network directly instead of converting to zp_consensus::Network where there were conversions

* fixes a bad merge (removes a network conversion in zcash_note_encryption)

* Adds a test for the Parameters impl for zebra_chain::Network

* fixes doc links

* - Makes the `activation_heights` config field optional by adding a #[serde(default)]
- Panics if a non-zero activation height is provided for the `Genesis` network upgrade
- Always sets the `Genesis` and `BeforeOverwinter` network upgrade activation heights to 0 and 1, `BeforeOverwinter` could be overwritten by a later network upgrade
- Makes the `activation_heights` field on `Parameters` private, adds/uses an accessor method instead, and adds a builder struct and `build()` method

* small refactor of activation_heights() method

* check that activation heights are in the right order

* Updates `NetworkUpgrade::activation_height()` to return the height of the next NetworkUpgrade if it doesn't find the activation height of `&self`

* checks that the miner address is of TestnetKind on Regtest and update assertion message

* Adds a DNetworkUpgradeActivationHeights struct for better control over how activation heights can be configured

* moves all ordered network upgrades to a constant, adds a no_duplicates test, moves struct with activation heights outside deserialization impl and accepts it in `ParametersBuilder::activation_heights()` instead of a Vec

* panics if any network upgrades are configured to activate at Height(0) because it's reserved for Genesis

* Simplifies the `ParametersBuilder::activation_heights()` method and removes an unnecessary test.

* Adds Sapling HRPs as fields on testnet params. (#8398)

* Applies suggestions from code review.

* Update zebra-chain/src/parameters/network_upgrade.rs

Co-authored-by: Marek <mail@marek.onl>

* Adds `network_name` field and accessor method on `Parameters`, uses it in the `Display` impl for `Network`

* Removes unnecessary `.filter()`

* adds constraints on valid network names and a test

* adds config field for setting network name, adds "with_" prefix to ParameterBuilder setter methods

* Adds `MainnetKind`, `TestnetKind`, and `RegtestKind` to reserved network names

* updates stored test configs and fixes `network_name()` docs

---------

Co-authored-by: Marek <mail@marek.onl>
2024-04-24 11:03:37 +00:00
Arya f8c153991b
add(consensus): Add `activation_heights` field to `NetworkParameters` and implement `Parameters` for `Network` (#8383)
* minor cleanup and rename

* Adds an empty NetworkParameters struct to Network::Testnet variant, updates production code.

* Updates tests

* Adds `NetworkKind` and uses it instead of `Network` in `HistoryTreeParts` and `transparent::Address`

* Adds a [network.testnet_parameters] section to the config, uses `NetworkKind` as zebra_network::Config::network field type, and converts 'Network' to `NetworkKind` before serializing

* Applies some suggestions from code review

* Applies suggestions from code review

* returns b58 prefix constants directly to remove From<NetworkKind> impl for zcash_primitives::consensus::Network

* Applies more suggestions from code review.

* moves conversions to zcash_primitives::consensus::Network to where they're used.

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

* rename `network` variables and method names typed as NetworkKind to `network_kind`

* use only test block heights for the network associated with them

* Applies more suggestions from code review.

* Rename `NetworkParameters` to `Parameters` and move it a new `testnet` module

* adds activation heights field

* updates stored test config, adds a quicker test for checking that stored configs can be parsed, adds an intermediate representation of activation heights

* implement Parameters for Network

* Passes &Network directly instead of converting to zp_consensus::Network where there were conversions

* fixes a bad merge (removes a network conversion in zcash_note_encryption)

* Adds a test for the Parameters impl for zebra_chain::Network

* fixes doc links

* - Makes the `activation_heights` config field optional by adding a #[serde(default)]
- Panics if a non-zero activation height is provided for the `Genesis` network upgrade
- Always sets the `Genesis` and `BeforeOverwinter` network upgrade activation heights to 0 and 1, `BeforeOverwinter` could be overwritten by a later network upgrade
- Makes the `activation_heights` field on `Parameters` private, adds/uses an accessor method instead, and adds a builder struct and `build()` method

* small refactor of activation_heights() method

* check that activation heights are in the right order

* Updates `NetworkUpgrade::activation_height()` to return the height of the next NetworkUpgrade if it doesn't find the activation height of `&self`

* checks that the miner address is of TestnetKind on Regtest and update assertion message

* Adds a DNetworkUpgradeActivationHeights struct for better control over how activation heights can be configured

* moves all ordered network upgrades to a constant, adds a no_duplicates test, moves struct with activation heights outside deserialization impl and accepts it in `ParametersBuilder::activation_heights()` instead of a Vec

* panics if any network upgrades are configured to activate at Height(0) because it's reserved for Genesis

* Simplifies the `ParametersBuilder::activation_heights()` method and removes an unnecessary test.

* Adds Sapling HRPs as fields on testnet params. (#8398)

* Applies suggestions from code review.

* Update zebra-chain/src/parameters/network_upgrade.rs

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2024-04-19 01:20:34 +00:00
Alfredo Garcia 887d4a2457
fix(beta-toolchain): Warnings (#8404)
* fix beta imports and other warnings

* clippy lints beta

* move attributes to after docs

Co-authored-by: Marek <mail@marek.onl>

* move some code to test module

* implement display instead of direct tostring for LongPollId

Co-authored-by: Arya <aryasolhi@gmail.com>

* fix typo

---------

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
2024-04-18 01:45:22 +00:00
Arya 16a39f89d0
add(consensus/network): Add an empty `Parameters` struct in `Network::Testnet` (#8368)
* minor cleanup and rename

* Adds an empty NetworkParameters struct to Network::Testnet variant, updates production code.

* Updates tests

* Adds `NetworkKind` and uses it instead of `Network` in `HistoryTreeParts` and `transparent::Address`

* Adds a [network.testnet_parameters] section to the config, uses `NetworkKind` as zebra_network::Config::network field type, and converts 'Network' to `NetworkKind` before serializing

* Applies some suggestions from code review

* Applies suggestions from code review

* returns b58 prefix constants directly to remove From<NetworkKind> impl for zcash_primitives::consensus::Network

* Applies more suggestions from code review.

* moves conversions to zcash_primitives::consensus::Network to where they're used.

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

* rename `network` variables and method names typed as NetworkKind to `network_kind`

* use only test block heights for the network associated with them

* Applies more suggestions from code review.

* Rename `NetworkParameters` to `Parameters` and move it a new `testnet` module

---------

Co-authored-by: Marek <mail@marek.onl>
2024-04-17 02:20:34 +00:00
idky137 4579722833
change(chain): Remove `Copy` trait impl from `Network` (#8354)
* removed derive copy from network, address and ledgerstate

* changed is_max_block_time_enforced to accept ref

* changed NetworkUpgrade::Current to accept ref

* changed NetworkUpgrade::Next to accept ref

* changed NetworkUpgrade::IsActivationHeight to accept ref

* changed NetworkUpgrade::TargetSpacingForHeight to accept ref

* changed NetworkUpgrade::TargetSpacings to accept ref

* changed NetworkUpgrade::MinimumDifficultySpacing_forHeight to accept ref

* changed NetworkUpgrade::IsTestnetMinDifficultyBlock to accept ref

* changed NetworkUpgrade::AveragingWindowTimespanForHeight to accept ref

* changed NetworkUpgrade::ActivationHeight to accept ref

* changed sapling_activation_height to accept ref

* fixed lifetime for target_spacings

* fixed sapling_activation_height

* changed transaction_to_fake_v5 and fake_v5_transactions_for_network to accept ref to network

* changed Input::vec_strategy to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_history.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/viewing_key* to accept ref to network

* changed functions in zebra-chain/src/transparent/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_primitives.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_note_encryption.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/history_tree* to accept ref to network

* changed functions in zebra-chain/src/block* to accept ref to network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* fixed errors in zebra-chain/src/block/arbitrary.rs

* finished fixing errors in zebra-chain - all crate tests pass

* changed functions in zebra-state::service::finalized_state to accept &Network

* changed functions in zebra-state::service::non_finalized_state to accept &Network

* zebra-state tests run but fail with overflow error

* zebra-state tests all pass

* converted zebra-network -- all crate tests pass

* applied all requested changes from review

* converted zebra-consensus -- all crate tests pass

* converted zebra-scan -- all crate tests pass

* converted zebra-rpc -- all crate tests pass

* converted zebra-grpc -- all crate tests pass

* converted zebrad -- all crate tests pass

* applied all requested changes from review

* fixed all clippy errors

* fixed build error in zebrad/src/components/mempool/crawler.rs
2024-03-19 20:45:27 +00:00
Arya 5e86ebb699
fix(scan): Improve gRPC method errors and add timeout to scan service to avoid hanging (#8318)
* updates ScanTask.subscribe() to return rsp_tx instead of awaiting it, returns service errors from grpc methods

* Adds a timeout to scan service and improves error message when subscribe responders are dropped

* Returns an error from `RegisterKeys` svc request when no keys are registered

* tests register keys error message

* Adds test for timeouts

* adds TODOs for fixing bugs

* Orders scan service requests in scan method

* removes CheckKeyHashes request

* fixes bad merge conflict resolution

* update test to check downcast to Elapsed error

* update errors in scan method

* update error msg in register_keys method

* fixes compiler error
2024-03-14 20:27:33 +00:00
Arya e64f3dcddd
add(test): Check that scan task removes keys, unify scan task acceptance tests, and add test to CI (#8308)
* merges scan task tests and checks that the scan task removes keys

* Adds test to CI

* Skip to expected block height and assert that the result is from that height

* Updates test docs and uses zebrad state path for scan cache

* Log "Zcash network: {network}" from scan task commands test
2024-03-04 21:25:50 +00:00
Arya 5a9281a7a8
fix(scan): Fix minor concurrency bug in the `scan` gRPC method (#8303)
* update scan task to notify subscribe caller once the new subscribed_keys are sent on the watch channel

* Fixes timing bug in scan gRPC method:

Joins register/subscribe scan service calls, sends SubscribeResults request first, and filters out duplicate results from channel

* Removes outdated TODO

* wraps subscribed_keys in an Arc before sending to watch channel, fixes typo

* Remove result senders for keys that have been removed
2024-02-21 23:29:13 +00:00
Arya 3929a526e5
add(scan): Implement SubscribeResults request for scan service (#8253)
* processes SubscribeResults messages

* send tx ids of results to the subscribe channel

* replaces BoxError with Report in scan_range

* adds a watch channel for using subscribed_keys in scan_range

* updates args to process_messages in test

* adds a `subscribe` method to ScanTask for sending a SubscribeResults cmd

* updates test for process_messages to cover subscribe cmds

* impls SubscribeResult service request and updates sender type

* adds test for SubscribeResults scan service request

* adds acceptance test

* updates tests and imports

* fixes acceptance test by using spawn_blocking to avoid blocking async executor and setting an appropriate start height

* fixes test

* Applies suggestions from code review.

* use tokio mpsc channel in scan task instead of std/blocking mpsc

* use tokio mpsc channel for results sender

* adds `was_parsed_keys_empty` instead of checking that all the parsed keys are new keys

* fixes test failures related to send errors in scan task

* returns height and key for scan results from subcribe_results results receiver

* hide scan_service mod in zebra-node-service behind feature
2024-02-13 00:42:40 +00:00
Marek 6b8cbf9904
add(scan): Implement `RegisterKeys` service request to register new keys (#8251)
* Refactor obtaining of activation heights

* Impl the `RegisterKeys` service request

* Mock viewing keys for tests

* Refactor tests

* Apply suggestions from code review

Co-authored-by: Arya <aryasolhi@gmail.com>

* Remove a redundant comment

I don't think we need to assume the genesis block doesn't contain
shielded data as the comment says.

* Avoid using a single-letter variable

* Refactor mocking Sapling scanning keys

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2024-02-09 15:23:19 +00:00
Arya 2c0bc3ac92
add(scan): Start scanner gRPC server with `zebrad` (#8241)
* adds clear_results RPC method for zebra-scan

* adds delete_keys rpc method

* adds docs

* Update zebra-grpc/proto/scanner.proto

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Apply suggestions from code review

* start zebra-scan gRPC server from zebrad start command

* adds a test that the scanner starts with zebrad

* adds a `listen_addr` field to the shielded scan config

* updates test to use a random port and set the listen_addr config field

* fixes test

* Update zebra-scan/src/config.rs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* fixes panic when trying to open multiple mutable storage instances.

* open db in blocking task

* fixes test

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2024-02-07 22:36:01 +00:00
Arya 2a004ffe9e
add(scan): Handle RegisterKeys messages in scan task (#8222)
* moves ScanTask to its own module in service module

* moves `process_messages()` method to scan_task.rs

* uses get_mut() and returns new keys

* updates types and adds scan_until() function

* adds and uses wait_for_height() function

* spawns scan_until_task_handler to poll scan_until_tasks FuturesUnordered

* corrects comment

* adds TODO

* updates expected test log, corrects panic msg, fixes fmt

* moves scan functions to scan_task module

* moves ScanTaskCommand and related method impls to its own module

* moves `ScanTask::mock()` test constructor to its own module

* creates `add_keys` mod and moves `scan_until()` function there

* moves scan task executor to its own module and adds ScanRangeTaskBuilder type

* renames add_keys to scan_range, moves executor to scan_task mod

* adds test for process_messages

* updates scan_height_and_store_results() to skip last_scanned_height check if there is no key height for a key in `parsed_keys`

* updates `default_test_config()`

* adds acceptance test for registering keys in ScanTask

* uses the right feature for the new acceptance test

* Applies suggestions from code review
2024-02-06 18:41:00 +00:00
teor 2ac6921d60
feat(mine): Add an internal Zcash miner to Zebra (#8136)
* Patch equihash to use the solver branch

* Add an internal-miner feature and set up its dependencies

* Remove 'Experimental' from mining RPC docs

* Fix a nightly clippy::question_mark lint

* Move a byte array utility function to zebra-chain

* fixup! Add an internal-miner feature and set up its dependencies

* Add an equihash::Solution::solve() method with difficulty checks

* Check solution is valid before returning it

* Add a TODO to check for peers before mining

* Move config validation into GetBlockTemplateRpcImpl::new()

* fixup! fixup! Add an internal-miner feature and set up its dependencies

* Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls

* Start adding an internal miner component

* Add the miner task to the start command

* Add basic miner code

* Split out a method to mine one block

* Spawn to a blocking thread

* Wait until a valid template is available

* Handle shutdown

* Run mining on low priority threads

* Ignore some invalid solutions

* Use a difference nonce for each solver thread

* Update TODOs

* Change the patch into a renamed dependency to simplify crate releases

* Clean up instrumentation and TODOs

* Make RPC instances cloneable and clean up generics

* Make LongPollId Copy so it's easier to use

* Add API to restart mining if there's a new block template

* Actually restart mining if there's a new block template

* Tidy instrumentation

* fixup! Move config validation into GetBlockTemplateRpcImpl::new()

* fixup! Make RPC instances cloneable and clean up generics

* Run the template generator and one miner concurrently

* Reduce logging

* Fix a bug in getblocktemplate RPC tip change detection

* Work around some watch channel change bugs

* Rate-limit template changes in the receiver

* Run one mining solver per available core

* Use updated C code with double-free protection

* Update to the latest solver branch

* Return and submit all valid solutions

* Document what INPUT_LENGTH means

* Fix watch channel change detection

* Don't return early when a mining task fails

* Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness

* Make existing parallelism docs and configs consistent

* Add a mining parallelism config

* Use the minimum of the configured or available threads for mining

* Ignore optional feature fields in tests

* Downgrade some frequent logs to debug

* Document new zebrad features and tasks

* Describe the internal-miner feature in the CHANGELOG

* Update dependency to de-duplicate equihash solutions

* Use futures::StreamExt instead of TryStreamExt

* Fix a panic message typo
2024-01-11 14:41:01 +00:00
Arya 81a3a2aacf
add(docs): Note default path to config in docs (#8143)
* Note default path to config file in README and docs

* Applies suggestions from review & adds new test config file

* removes default path for 'Other' OSes from ZebradConfig docs

* fixes config_tests
2024-01-10 06:53:32 +00:00
Alfredo Garcia cdfbecf5f5
avoid starting the scan task if no keys to scan are present (#8059) 2023-12-06 01:56:43 +00:00
teor cb9452c5e3
change(ci): When building crates individually, build all targets, and run clippy (#8024)
* Run clippy and build all targets on all crates individually

* Fix prod and test features for scanner deps

* Standardise dependency order

* Remove unnecessary async in tests

* Fix an unused import in a test

* Work around a no space left on device error

* Actually just use a larger runner
2023-11-30 00:51:20 +00:00
teor 1d241afbaa
change(state): Expose ZebraDb methods that can create different kinds of databases (#8002)
* Provide access to DiskDb and DiskWriteBatch outside the state using a feature

* Actually let's export ZebraDb for the format upgrade code

* Pass column families to ZebraDb as an argument

* Allow the database kind to be changed in config.rs

* Use the state kind in finalized_state.rs

* Allow different database kinds in ZebraDb, but don't move the upgrade code yet

* Allow different database kinds in DiskDb

* Allow different database kinds in upgrade.rs, but don't split the upgrade code out yet

* Add new arguments to raw database tests

* Fix doc links

* Fix internal imports

* Fix unused code

* Update zebrad version metadata

* Create a specific state database delete function

* Fix state exports

* Fix zebrad tests

* Fix zebrad state write tests

* Make CI run again

* Fix dead code warnings for test methods

* Remove unnecessary async on some tests

* Fix logging required by tests

* Fix logging required in test itself

* Fix variable names

* Try to copy the message and add regexes
2023-11-28 13:49:11 +00:00
teor af82a76d18
cleanup(clippy): Remove unnecessary `try_into()` with `cargo clippy --fix` (#7940)
* Remove unnecessary try_into() with cargo clippy --fix

* Manually remove some unnecessary try_from()s

* impl From<Diversifier> for pallas::Affine instead of TryFrom

* Remove unused imports

* cargo fmt --all

* Remove redundant comma
2023-11-14 20:31:49 +00:00
teor afbe807060
change(docs): Replace doc.zebra.zfnd.org API documentation with docs.rs (#7886)
* Simple replacements of doc.zebra.zfnd.org with docs.rs

* Manual fixes for specific main/internal/external docs

* Point developer docs to doc-internal.zebra.zfnd.org

* fastmod --glob '\!.git' -- doc.zebra.zfnd.org/zebrad docs.rs/zebrad/latest/zebrad

* Manually remove any remaining doc.zfnd.zebra.org links

* Remove the external docs job

* Add changelog entry and fix links

* Fix links that were broken before this PR
2023-11-01 22:09:40 +00:00
Alfredo Garcia f3048653c8
chore: fix typos (#7877)
Co-authored-by: Dimitris Apostolou <dimitris.apostolou@icloud.com>
2023-10-30 20:06:35 +00:00
Arya 0a3790b73e
change(consensus): Remove Sprout and Sapling parameter download task and debug_skip_preload config (#7844)
* removes groth16 download task

* updates docs.

* Adds new config to test configs

* fixes compatibility with past configs

* uses inner config to deserialize removed field for compatibility

* update docs

* updates latest config

* Applies suggestions from code review

* Avoid duplicating hard-coded default values

---------

Co-authored-by: teor <teor@riseup.net>
2023-10-27 06:12:57 +00:00
teor 64f777274c
fix(security): fix concurrency issues in tree key formats, and CPU usage in genesis tree roots (#7392)
* Add tree key format and cached root upgrades

* Document the changes in the upgrades

* Remove unnecessary clippy::unwrap_in_result

* Fix database type

* Bump state version

* Skip some checks if the database is empty

* Fix tests for a short state upgrade

* Disable format checks in some tests

* Document state performance issues

* Clarify upgrade behaviour

* Clarify panic messages

* Delete incorrect genesis trees write code

* Fix metrics handling for genesis

* Remove an unused import

* Explain why genesis anchors are ok

* Update snapshots

* Debug a failing test

* Fix some tests

* Fix missing imports

* Move the state check in a test

* Fix comment and doc typos

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>

* Clarify what a long upgrade is

* Rename unused function arguments

Co-authored-by: Marek <mail@marek.onl>

* Add all_unordered log regex matching methods

* Fix timing issues with version upgrades and other logs

* Fix argument name in docs

Co-authored-by: Marek <mail@marek.onl>

* Explain match until first for all regexes behaviour better

---------

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
2023-10-19 14:50:46 +00:00
teor 08ce2ad500
feature(mining): Enable mining RPCs by default in production builds (#7740)
* Enable getblocktemplate-rpcs in production builds

* Update mining instructions

* Consistently use ZF funding stream addresses in examples

* Simplify skip_getblocktemplate()

* Set the test miner address based on the network

* Correctly skip checks in lightwalletd send transactions test

* Wait longer for transaction query

* Wait longer before re-checking the mempool

* Skip mempool checks if it wasn't queried by lightwalletd

* rustfmt

* clippy and rustfmt

* Fix attribute issue

* Fix typo

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-10-18 04:15:17 +00:00
teor 2a077f21c7
fix(ci): Skip mempool checks if it wasn't queried by lightwalletd (#7744)
* Correctly skip checks in lightwalletd send transactions test

* Wait longer for transaction query

* Wait longer before re-checking the mempool

* Skip mempool checks if it wasn't queried by lightwalletd

* rustfmt

* clippy and rustfmt

* Fix attribute issue
2023-10-17 02:56:00 +00:00
teor ae52e3d23d
change(ui): Enable the progress bar feature by default, but only show progress bars when the config is enabled (#7615)
* Add a progress bar config that is disabled unless the feature is on

* Simplify the default config

* Enable the progress bar feature by default, but require the config

* Rename progress bars config to avoid merge conflicts

* Use a log file when the progress bar is activated

* Document how to configure progress bars

* Handle log files in config_tests and check config path

* Fix doc link

* Fix path check

* Fix config log matching

* Fix clippy warning

* Add tracing to config tests

* It's zebrad not zebra

* cargo fmt --all

* Update release for config file changes

* Fix config test failures

* Allow printing to stdout in a method
2023-10-12 00:25:37 +00:00
Arya 931e0a7587
fix(tests): Checking if transactions get into the mempool fails in lightwalletd tests (#7644)
* copy zcash/lightwalletd service.proto

* uses correct assert & minor cleanup

* adds comment about checking service.proto file when

* adds workaround in send_transaction_test

* Update docs

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: teor <teor@riseup.net>
2023-10-10 02:00:51 +00:00
teor 0faa0697d5
change(state): Only do continuous format checks in CI (#7627)
* Only do continuous format checks if enabled by a config

* Run continuous format checks in CI
2023-09-27 21:42:43 +00:00
Alfredo Garcia 09c1f994f0
remove intermittent asserts in test (#7600) 2023-09-21 03:20:55 +00:00
teor 7a7d79dfaf
fix(state): Use correct end heights for end of block subtrees during the full sync (#7566)
* Avoid manual handling of previous sapling trees by using iterator windows instead

* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead

* Simplify adding notes by using the exact number of remaining notes

* Simplify by skipping the first block, because it can't complete a subtree

* Re-use existing tree update code

* Apply the sapling changes to orchard subtree updates

* add a reverse database column family iterator function

* Make skipping the lowest tree independent of iteration order

* Move new subtree checks into the iterator, rename to end_height

* Split subtree calculation into a new method

* Split the calculate and write methods

* Quickly check the first subtree before running the full upgrade

* Do the quick checks every time Zebra runs, and refactor slow check error handling

* Do quick checks for orchard as well

* Make orchard tree upgrade match sapling upgrade code

* Upgrade subtrees in reverse height order

* Bump the database patch version so the upgrade runs again

* Reset previous subtree upgrade data before doing this one

* Add extra checks to subtree calculation to diagnose errors

* Use correct heights for subtrees completed at the end of a block

* Add even more checks to diagnose issues

* Instrument upgrade methods to improve diagnostics

* Prevent modification of re-used trees

* Debug with subtree positions as well

* Fix an off-by-one error with completed subtrees

* Fix typos and confusing comments

Co-authored-by: Marek <mail@marek.onl>

* Fix mistaken previous tree handling and end tree comments

* Remove unnecessary subtraction in remaining leaves calc

* Log heights when assertions fail

* Fix new subtree detection filter

* Move new subtree check into a method, cleanup unused code

* Remove redundant assertions

* Wait for subtree upgrade before testing RPCs

* Fix subtree search in quick check

* Temporarily upgrade subtrees in forward height order

* Clarify some comments

* Fix missing test imports

* Fix subtree logging

* Add a comment about a potential hang with future upgrades

* Fix zebrad var ownership

* Log more info when add_subtrees.rs fails

* cargo fmt --all

* Fix unrelated clippy::unnecessary_unwrap

* cargo clippy --fix --all-features --all-targets; cargo fmt --all

* Stop the quick check depending on tree de-duplication

* Refactor waiting for the upgrade into functions

* Wait for state upgrades whenever the cached state is updated

* Wait for the testnet upgrade in the right place

* Fix unused variable

* Fix a subtree detection bug and comments

* Remove an early reference to reverse direction

* Stop skipping subtrees completed at the end of blocks

* Actually fix new subtree code

---------

Co-authored-by: Marek <mail@marek.onl>
2023-09-19 14:49:36 +00:00
Alfredo Garcia 91ec8b910b
change(tests): update tests for new lightwalletd version used in Zebra (#7349)
* update docs for new lightwalletd version used in Zebra

* update some asserts

* add `RUST_LOG` to docker `entrypoint.sh` file

* clarify zecwallet-cli is untested with zcash/lightwalletd fork in the book

* revert entrypoint rust log changes

* remove sleep

* revert doc changes

* change formatting in expected output

* try sleep now that CI is working

* try sending more transactions

* remove tests with failures

* remove newline
2023-09-14 05:25:42 +00:00