Commit Graph

593 Commits

Author SHA1 Message Date
Arya 0ef9987e9e
fix(state): Write database format version to disk atomically to avoid a rare panic (#8795)
* Splits `atomic_write_to_tmp_file` out of `zebra_network::Config::update_peer_cache`

* Uses the new `atomic_write_to_tmp_file` fn in `update_peer_cache()`

* Replaces repetitive code for getting the default peer and state cache directories with `default_cache_dir()`

* Converts `atomic_write_to_tmp_file` to a blocking function and adds `spawn_atomic_write_to_tmp_file` for use in async environments.

* Uses `atomic_write_to_tmp_file` to write database versions to disk

* Removes `spawn_atomic_write_to_tmp_file()` and inlines its body at its callsite to avoid adding tokio as a dependency of zebra-chain.

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2024-08-29 21:09:27 +00:00
Arya 83c6725e84
change(rpc): Add value pool balances to `getblockchaininfo` RPC method response (#8769)
* Moves `Zec` type out from behind feature flag

* Adds 'ValuePoolBalance` type

* Updates getblockchaininfo return type to a BoxFuture

* minor refactor

* Adds service request

* Adds real value balances to getblockchaininfo RPC response

* Updates snapshots and the suggested command for updating snapshots

* Uses generic error constructors wherever possible and removes outdated TODOs

* Updates prop tests to handle mock service requests
2024-08-16 19:54:44 +00:00
Marek 88f9bffec9
change(state): Support in-place disk format upgrades for major database version bumps (#8748)
* Reuse existing db after a major upgrade

* Don't delete dbs that can be reused

* Apply suggestions from code review

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Fix formatting

* Create only the parent dir for the db

---------

Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2024-08-09 15:39:38 +00:00
Marek 82ded59a31
change: Track the balance of the deferred chain value pool (#8729)
* Addresses clippy lints

* checks network magic and returns early from `is_regtest()`

* Moves  `subsidy.rs` to `zebra-chain`, refactors funding streams into structs, splits them into pre/post NU6 funding streams, and adds them as a field on `testnet::Parameters`

* Replaces Vec with HashMap, adds `ConfiguredFundingStreams` type and conversion logic with constraints.

Minor refactors

* Empties recipients list

* Adds a comment on num_addresses calculation being invalid for configured Testnets, but that being okay since configured testnet parameters are checked when they're being built

* Documentation fixes, minor cleanup, renames a test, adds TODOs, and fixes test logic

* Removes unnecessary `ParameterSubsidy` impl for &Network, adds docs and TODOs

* Adds a "deferred" FundingStreamReceiver, adds a post-NU6 funding streams, updates the `miner_fees_are_valid()` and `subsidy_is_valid()` functions to check that the deferred pool contribution is valid and that there are no unclaimed block subsidies after NU6 activation, and adds some TODOs

* adds `lockbox_input_value()` fn

* Adds TODOs for linking to relevant ZIPs and updating height ranges

* Adds `nu6_lockbox_funding_stream` acceptance test

* updates funding stream values test to check post-NU6 funding streams too, adds Mainnet/Testnet NU6 activation heights, fixes lints/compilation issue

* Reverts Mainnet/Testnet NU6 activation height definitions, updates `test_funding_stream_values()` to use a configured testnet with the post-NU6 Mainnet funding streams height range

* reverts unnecessary refactor

* appease clippy

* Adds a test for `lockbox_input_value()`

* Applies suggestions from code review

* Fixes potential panic

* Fixes bad merge

* Update zebra-chain/src/parameters/network_upgrade.rs

* Updates acceptance test to check that invalid blocks are rejected

* Checks that the original valid block template at height 2 is accepted as a block submission

* Reverts changes for coinbase should balance exactly ZIP

* Add `Deferred` to `ValueBalance`

* Update snapshots

* Unrelated: Revise docs

* Add TODOs

* Stop recalculating the block subsidy

* Track deferred balances

* Support heights below slow start shift in halvings

* Fix `CheckpointVerifiedBlock` conversion in tests

* Allow deserialization of legacy `ValueBalance`s

* Simplify docs

* Fix warnings raised by Clippy

* Fix warnings raised by `cargo fmt`

* Update zebra-chain/src/block.rs

Co-authored-by: Arya <aryasolhi@gmail.com>

* Refactor docs around chain value pool changes

* updates test name

* Updates deferred pool funding stream name to "Lockbox", moves post-NU6 height ranges to constants, updates TODO

* Updates `get_block_subsidy()` RPC method to exclude lockbox funding stream from `fundingstreams` field

* Adds a TODO for updating `FundingStreamReceiver::name()` method docs

* Updates `FundingStreamRecipient::new()` to accept an iterator of items instead of an option of an iterator, updates a comment quoting the coinbase transaction balance consensus rule to note that the current code is inconsistent with the protocol spec, adds a TODO for updating the quote there once the protocol spec has been updated.

* Update zebra-consensus/src/checkpoint.rs

Co-authored-by: Arya <aryasolhi@gmail.com>

* Update docs for value balances

* Cleanup: Simplify getting info for FS receivers

* Avoid a panic when deserializing value balances

* Uses FPF Testnet address for post-NU6 testnet funding streams

* Updates the NU6 consensus branch id

* Update zebra-consensus/src/checkpoint.rs

* Bump the major database format version

* Add a database upgrade mark

* Fix tests after merge

* trigger GitHub actions

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
2024-08-05 20:30:47 +00:00
Marek 16168d7623
Store Sprout trees with cached roots (#8741)
Zebra's database expects all stored trees to have their roots cached in
them.
2024-08-03 22:29:09 +00:00
Arya 7ee9e2b25c
Fixes cargo check annotations on Windows (#8738) 2024-08-03 00:35:10 +00:00
Marek 45261a26eb
Don't use a fixed size of serialized ValueBalance (#8732) 2024-07-31 16:09:24 +00:00
Arya 6181ebe239
Addresses clippy lints (#8723) 2024-07-26 14:29:35 +00:00
Arya 4213e82a4f
change(rpc): Adds a TrustedChainSync struct for keeping up with Zebra's non-finalized best chain from a separate process (#8596)
* Adds an init_read_only() fn in zebra-state

* moves elasticsearch initialization to `FinalizedState::new_with_debug()`

* Updates callers of `FinalizedState::{new, new_with_debug}` to pass a bool to try enabling elasticsearch

* Adds a non-finalized read state syncer to zebra-rpc

* moves, removes, updates, or addresses TODOs

* reduces disk IO while waiting for the a new chain tip & updates the chain tip sender when the finalized tip has changed.

* Returns boxed errors from RpcRequestClient methods instead of color_eyre type

* Avoids resetting the non-finalized state when there's an error getting a block unless it has the missing block error code.

* Adds stub for acceptance test(s) and removes outdated TODO

* adds TODOs for testing

* Tests that `ChainTipChange` is updated when the non-finalized best chain grows

* adds a last_chain_tip_hash and uses a FuturesOrdered for getblock requests

* Fixes a pre-flush sync issue by using a secondary db instead of a read-only db

* Moves disk IO to blocking tasks

* Updates acceptance test to how forks are handled

* Checks synced read state for all of the expected blocks

* checks that there isn't a tip change until the best chain changes

* checks for chain tip changes in test

* run test without feature

* fixes lint

* Fixes compilation/test issues

* Adds docs / comments, moves HexData out from behind the getblocktemplate-rpcs feature flag, moves test behind the mining feature flag.

* Fixes lints

* removes syncer and rpc-syncer features

* Fixes test on Windows, applies suggestions from code review

* Updates `POLL_DELAY` documentation

* Updates method docs

* Fixes a test bug

* use rpc-client feature in zebrad production code

* use rpc-client feature in zebra-node-services for building zebra-rpc crate

---------

Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2024-07-09 14:15:47 +00:00
Arya 1b5f9426f9
change(state): Add an init function for a standalone `ReadStateService` (#8595)
* Adds an init_read_only() fn in zebra-state

* moves elasticsearch initialization to `FinalizedState::new_with_debug()`

* Updates callers of `FinalizedState::{new, new_with_debug}` to pass a bool to try enabling elasticsearch

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2024-06-12 13:35:13 +00:00
Arya 5ca40aa58b
add(network): Adds a network magic field to `testnet::Parameters` and the config (#8524)
* Adds a `target_difficulty_limit` field on `testnet::Parameters`

* updates test to increment block nonce until finding a block that's below the difficulty threshold

* increment the nonce while the difficulty is invalid instead of while the difficulty threshold is invalid

* Adds comments

* moves network Magic type from zebra-network to zebra-chain

* Adds `network_magic` field to `testnet::Parameters` and uses the regtest network magic from zcashd

* Add a network magic config field for custom testnets

* Adds/updates tests

* Update zebra-chain/src/parameters/network/testnet.rs

* Adds a link to the Regtest network magic in zcashd

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2024-05-18 20:30:59 +00:00
Arya 0040c2be87
fix(consensus): Update `median_timespan()` method to align with zcashd implementation (#8491)
* Updates `median_timespan()` method to align with zcashd implementation of ActualTimespan

* Minor refactor

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2024-05-07 20:51:13 +00:00
Arya 8a786fe6ce
add(consensus): Adds block construction and validation for NU5 from block height 1 for Regtest (#8475)
* Always activate Canopy at Height(1) on Regtest

* Ignores the zip 212 grace period on configured Testnets and Regtest

* - Returns early when there is no Heartwood activation height when creating or updating HistoryTree
- Skips call to `check::legacy_chain()` when no NU5 activation height is set (it would return immediately anyway)
- Replaces `.map()` with `.filter_map()` in `NetworkUpgrade::target_spacings()`
- Removes outdated TODO

* - When proof of work is disabled, skips checking if Zebra is synced to the network tip in the getblocktemplate method

* Sets full_verifier_utxo_lookahead to Height::MIN instead of panicking

* When network is regtest, skips starting sync task and commits the genesis block if it's missing in the state

* updates/fixes test config

* Adds test for committing Canopy blocks on Regtest

* - Updates median time past and difficulty checks to use fewer than 11/17/28 blocks

* uses SLOW_START_INTERVAL of 0 if PoW is disabled, adds TODOs

* Update getblocktemplate method to return reserved chain history activation root hash, uses Nu5 at height 1

Test passes.

* Updates test to expect NU5 as the default nu activation at Height(1)

* Removes invalid difficulty snapshot

* fixes tests

* removes regtest NU5 activation height config field

* Apply suggestions from code review

* Update zebra-state/src/service/check/difficulty.rs

Co-authored-by: Marek <mail@marek.onl>

* Update zebra-state/src/service/check/difficulty.rs

* Updates docs/comments, renames an argument

---------

Co-authored-by: Marek <mail@marek.onl>
2024-05-02 19:43:42 +00:00
Arya c28e37bf23
disables `rejection_restores_internal_state_genesis` test on Windows (#8468) 2024-04-26 05:30:31 +00:00
Alfredo Garcia 887d4a2457
fix(beta-toolchain): Warnings (#8404)
* fix beta imports and other warnings

* clippy lints beta

* move attributes to after docs

Co-authored-by: Marek <mail@marek.onl>

* move some code to test module

* implement display instead of direct tostring for LongPollId

Co-authored-by: Arya <aryasolhi@gmail.com>

* fix typo

---------

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
2024-04-18 01:45:22 +00:00
Alfredo Garcia 2995ea087b
send ping to ES server before inserting data (#8409) 2024-04-17 19:09:16 +00:00
Arya 16a39f89d0
add(consensus/network): Add an empty `Parameters` struct in `Network::Testnet` (#8368)
* minor cleanup and rename

* Adds an empty NetworkParameters struct to Network::Testnet variant, updates production code.

* Updates tests

* Adds `NetworkKind` and uses it instead of `Network` in `HistoryTreeParts` and `transparent::Address`

* Adds a [network.testnet_parameters] section to the config, uses `NetworkKind` as zebra_network::Config::network field type, and converts 'Network' to `NetworkKind` before serializing

* Applies some suggestions from code review

* Applies suggestions from code review

* returns b58 prefix constants directly to remove From<NetworkKind> impl for zcash_primitives::consensus::Network

* Applies more suggestions from code review.

* moves conversions to zcash_primitives::consensus::Network to where they're used.

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

* rename `network` variables and method names typed as NetworkKind to `network_kind`

* use only test block heights for the network associated with them

* Applies more suggestions from code review.

* Rename `NetworkParameters` to `Parameters` and move it a new `testnet` module

---------

Co-authored-by: Marek <mail@marek.onl>
2024-04-17 02:20:34 +00:00
Arya ad34585590
add(rpc): Add block times to verbose output of `getblock` RPC method (#8384)
* Returns block times from the getblock RPC when used with verbosity = 1 (it's already included with verbosity = 0 but this makes it easier to use).

* cleanup/refactor, adds MapServerError and OkOrServerError traits

* moves rpc error conversion traits to their own module

* Only returns block time for verbosity = 2, updates snapshots
2024-04-12 20:07:44 +00:00
Alfredo Garcia 339a42f86d
change database logs to human readable forms (#8389) 2024-04-11 10:41:52 +00:00
Elijah Hampton 49fca309cf
Adds logging of column family size and database size on startup and s… (#8336)
* Adds logging of column family size and database size on startup and shutdown

* Changes log level of column families size strings to debug. Adds TODO comment to use human_bytes crate for human-readable format of metrics. Adds print_db_metrics function to ZebraDb struct.

* Calls enumerate() on column_families to access index var

* Resolves cargo fmt checker results

* Resolves clippy lint

* Runs and fixes changes from fmt, clippy, check and test

* Removes prop.txt

* minor doc changes

---------

Co-authored-by: Elijah Hampton <elijahhampton@MBP-de-Elijah-2.lan>
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2024-03-29 00:00:28 +00:00
idky137 4579722833
change(chain): Remove `Copy` trait impl from `Network` (#8354)
* removed derive copy from network, address and ledgerstate

* changed is_max_block_time_enforced to accept ref

* changed NetworkUpgrade::Current to accept ref

* changed NetworkUpgrade::Next to accept ref

* changed NetworkUpgrade::IsActivationHeight to accept ref

* changed NetworkUpgrade::TargetSpacingForHeight to accept ref

* changed NetworkUpgrade::TargetSpacings to accept ref

* changed NetworkUpgrade::MinimumDifficultySpacing_forHeight to accept ref

* changed NetworkUpgrade::IsTestnetMinDifficultyBlock to accept ref

* changed NetworkUpgrade::AveragingWindowTimespanForHeight to accept ref

* changed NetworkUpgrade::ActivationHeight to accept ref

* changed sapling_activation_height to accept ref

* fixed lifetime for target_spacings

* fixed sapling_activation_height

* changed transaction_to_fake_v5 and fake_v5_transactions_for_network to accept ref to network

* changed Input::vec_strategy to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_history.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/viewing_key* to accept ref to network

* changed functions in zebra-chain/src/transparent/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_primitives.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_note_encryption.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/history_tree* to accept ref to network

* changed functions in zebra-chain/src/block* to accept ref to network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* fixed errors in zebra-chain/src/block/arbitrary.rs

* finished fixing errors in zebra-chain - all crate tests pass

* changed functions in zebra-state::service::finalized_state to accept &Network

* changed functions in zebra-state::service::non_finalized_state to accept &Network

* zebra-state tests run but fail with overflow error

* zebra-state tests all pass

* converted zebra-network -- all crate tests pass

* applied all requested changes from review

* converted zebra-consensus -- all crate tests pass

* converted zebra-scan -- all crate tests pass

* converted zebra-rpc -- all crate tests pass

* converted zebra-grpc -- all crate tests pass

* converted zebrad -- all crate tests pass

* applied all requested changes from review

* fixed all clippy errors

* fixed build error in zebrad/src/components/mempool/crawler.rs
2024-03-19 20:45:27 +00:00
Za Wilcox 3bef54b764
change(consensus): Refactor production code for network consensus rules to `Network` methods (#8340)
* begin refactor suggested as "step 2": https://github.com/ZcashFoundation/zebra/issues/7968#issue-2003245309
Squashed from multiple commits to enable partial rebase

* break out more little traits

* add activation implementation leveraging From<Network> for lrz::cons::

* for transfer of ownership I cannot return a type that's owned by the method

* hrp_sapling_extended_full_viewing_key

* complete implementation of interface of Parameters on Network reuse Parameters on zcash Network where possible

* move doc-comments to trait declarations (from impls)

* Simplify/complete Parameters impl for Network

* Add checkpoint_list method, move documentation, etc

* move last match network to inside network method

* add back comment

* use zcash_address for parameter types in zebra-chain

* use inherent methods instead of big parameters passthrough

* revert to implementation of From on zcash_primitives::..::Network vs &zcash_prim...

* move match

* add test to block maximum time rule

* update changelog

* finish porting target_difficutly_limit

* remove obscelete code comment

* revert non-functional change

* finish migrating target_difficulty_limit, checkpoint_list

* update changelog

---------

Co-authored-by: Hazel OHearn <gygaxis@zingolabs.org>
2024-03-12 21:41:44 +00:00
idky137 9b91d4bc0e
change(tests): Remove Matches on Network From Tests (#8295)
* added functions for fetching block vectors

* inserted new network methods for vector fetching into zebra-chain

* changed tag back to test/feature=branch

* changed tag back to test/feature=branch

* changed tag back to test/feature=branch

* changed feature tag to proptest-impl, started implementing in zebra-consensus tests

* adding methods to zebra-consensus, lifetime error in src/transaction/tests.rs, needs refactoring

* finished adding methods to zebra-consensus

* finished adding new methods to zebrad

* added new methods to zebra-rpc and zebra-scan

* finished removing statements matching on Network from tests

* updated new error message

* changed get_block_bytes() and get_block_sapling_roots_bytes to return option and removed serialization error types as per requested changes in PR review

* removed match statements from zebra_chain::transaction::arbitrary::test_transactions() and new zebra-grpc tests

* moved zebra-chain::test_utils to zebra-chain::test

* removed get_ prefix from getter methods

* renamed zebra-chain::test to zebra-chain::tests

* renamed zebra-chain::test to zebra-chain::tests

* fixed clippy warnings

* changed block_map to return clone
2024-03-05 09:12:25 -05:00
Marek 130c4e95e8
Fix formatting in `proptest` macros (#8252) 2024-02-09 01:54:54 +00:00
teor 2ac6921d60
feat(mine): Add an internal Zcash miner to Zebra (#8136)
* Patch equihash to use the solver branch

* Add an internal-miner feature and set up its dependencies

* Remove 'Experimental' from mining RPC docs

* Fix a nightly clippy::question_mark lint

* Move a byte array utility function to zebra-chain

* fixup! Add an internal-miner feature and set up its dependencies

* Add an equihash::Solution::solve() method with difficulty checks

* Check solution is valid before returning it

* Add a TODO to check for peers before mining

* Move config validation into GetBlockTemplateRpcImpl::new()

* fixup! fixup! Add an internal-miner feature and set up its dependencies

* Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls

* Start adding an internal miner component

* Add the miner task to the start command

* Add basic miner code

* Split out a method to mine one block

* Spawn to a blocking thread

* Wait until a valid template is available

* Handle shutdown

* Run mining on low priority threads

* Ignore some invalid solutions

* Use a difference nonce for each solver thread

* Update TODOs

* Change the patch into a renamed dependency to simplify crate releases

* Clean up instrumentation and TODOs

* Make RPC instances cloneable and clean up generics

* Make LongPollId Copy so it's easier to use

* Add API to restart mining if there's a new block template

* Actually restart mining if there's a new block template

* Tidy instrumentation

* fixup! Move config validation into GetBlockTemplateRpcImpl::new()

* fixup! Make RPC instances cloneable and clean up generics

* Run the template generator and one miner concurrently

* Reduce logging

* Fix a bug in getblocktemplate RPC tip change detection

* Work around some watch channel change bugs

* Rate-limit template changes in the receiver

* Run one mining solver per available core

* Use updated C code with double-free protection

* Update to the latest solver branch

* Return and submit all valid solutions

* Document what INPUT_LENGTH means

* Fix watch channel change detection

* Don't return early when a mining task fails

* Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness

* Make existing parallelism docs and configs consistent

* Add a mining parallelism config

* Use the minimum of the configured or available threads for mining

* Ignore optional feature fields in tests

* Downgrade some frequent logs to debug

* Document new zebrad features and tasks

* Describe the internal-miner feature in the CHANGELOG

* Update dependency to de-duplicate equihash solutions

* Use futures::StreamExt instead of TryStreamExt

* Fix a panic message typo
2024-01-11 14:41:01 +00:00
bishopcheckmate 07693a1596
refactor: remove obsolete clippy allows in non_finalized_state (#8140) 2024-01-09 01:06:26 +00:00
dependabot[bot] f21d7c6934
build(deps): bump the prod group with 6 updates (#8125)
* build(deps): bump the prod group with 6 updates

Bumps the prod group with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [futures](https://github.com/rust-lang/futures-rs) | `0.3.29` | `0.3.30` |
| [tokio](https://github.com/tokio-rs/tokio) | `1.35.0` | `1.35.1` |
| [metrics](https://github.com/metrics-rs/metrics) | `0.21.1` | `0.22.0` |
| [metrics-exporter-prometheus](https://github.com/metrics-rs/metrics) | `0.12.2` | `0.13.0` |
| [reqwest](https://github.com/seanmonstar/reqwest) | `0.11.22` | `0.11.23` |
| [owo-colors](https://github.com/jam1garner/owo-colors) | `3.5.0` | `4.0.0` |


Updates `futures` from 0.3.29 to 0.3.30
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.29...0.3.30)

Updates `tokio` from 1.35.0 to 1.35.1
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.35.0...tokio-1.35.1)

Updates `metrics` from 0.21.1 to 0.22.0
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/compare/metrics-v0.21.1...metrics-v0.22.0)

Updates `metrics-exporter-prometheus` from 0.12.2 to 0.13.0
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/compare/metrics-exporter-prometheus-v0.12.2...metrics-exporter-prometheus-v0.13.0)

Updates `reqwest` from 0.11.22 to 0.11.23
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.22...v0.11.23)

Updates `owo-colors` from 3.5.0 to 4.0.0
- [Commits](https://github.com/jam1garner/owo-colors/compare/v3.5.0...v4.0.0)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: metrics
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: metrics-exporter-prometheus
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: owo-colors
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>

* update all metric macros

* fix deprecated function

* fix duplicated deps

* Fix an incorrect gauge method call

* Expand documentation and error messages for best chain length

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2024-01-02 01:26:54 +00:00
Alfredo Garcia c0f88ede8e
make bulk size the same for testnet and mainnet (#8127) 2024-01-01 23:21:22 +00:00
teor ad015e04d9
Move history tree and value balance to typed column families (#8115)
* impl TryFrom<zcash_primitives::BlockHeight> for Height

* Add type-safe read and write database methods

* Only allow typed access to the scanner DB

* Update docs

* Implement a common method as a trait

* Fix imports

* Tidy state imports

* Activate tracing logging macros in the whole scanner crate

* Fix dead code warnings

* Use a more sensible export order

* Remove a 1.72 lint exception now 1.74 is stable

* Switch history trees over to TypedColumnFamily, and remove redundant code

* Add typed batch creation methods, and switch history trees to them

* Convert ValueBalance to typed column families

* Make the APIs compatible after a merge

* Use `ZebraDb` instead of `DiskDb` where needed

---------

Co-authored-by: Marek <mail@marek.onl>
2023-12-20 23:20:57 +00:00
teor 4b5838c500
Make sure scanner database is accessed using the correct types (#8112)
* impl TryFrom<zcash_primitives::BlockHeight> for Height

* Add type-safe read and write database methods

* Only allow typed access to the scanner DB

* Update docs

* Implement a common method as a trait

* Fix imports

* Tidy state imports

* Activate tracing logging macros in the whole scanner crate

* Fix dead code warnings
2023-12-18 16:33:49 +00:00
Marek 22852bc81d
change(state): Allow opening the database in a read-only mode (#8079)
* Allow opening the database read-only mode

* Update zebra-scan/src/storage/db.rs

Co-authored-by: teor <teor@riseup.net>

* Refactor skipping database upgrades

* Fix errors caused by the merge

* Add new argument in new_test_storage()

* Simplify test storage code

* Fix importing the `new_test_storate` fn

---------

Co-authored-by: teor <teor@riseup.net>
2023-12-13 21:54:00 +00:00
Alfredo Garcia 92758a0d9f
feat(scanner): Restart scanning where left (#8080)
* start scanner where it was left

* fix tests

* add a `scan_start_where_left` test

* refactor a log msg

* fix some comments

* remove function

* fix doc comment

* clippy

* fix `sapling_keys_and_last_scanned_heights()`

* simplify start height

* i went too far, revert some changes back

* change log info to every 10k blocks

* fix build

* Update height snapshot code and check last height is consistent

* Add strictly before and strictly after database key gets

* Move to the previous key using strictly before ops

* Assert that keys are only inserted once

* Update the index in each loop

* Update snapshots

* Remove debugging code

* start scanning at min available height

---------

Co-authored-by: teor <teor@riseup.net>
2023-12-13 20:16:26 +00:00
teor 9ace6f8a4e
test(scan): Add typed database format snapshots to the scanner (#8083)
* Refactor test data into functions

* Add a typed snapshot test for scanner storage

* Use standard hex serialization with SaplingScannedResult

* Simplify transaction::Hash hex serialization

* Sort HashMaps before snapshotting

* Add typed snapshot data
2023-12-12 22:34:59 +00:00
teor 3318eaaa22
test(scan): Add raw database format snapshots to the scanner (#8075)
* Make some scanner storage methods more flexible

* Move tests to a submodule and expose test functions and constants

* Make scanner functions clearer and easier to use

* Simplify state snapshot test code

* Add raw data snapshot tests for the scanner

* Add snapshots

* Fix import path

* Fix import conditional compilation

* fix imports

* fix imports 2

* Put read and write db exports together

* Remove confusing IntoDisk/FromDisk impl

* Fix an incorrect unused method that could panic

* Delete a test that is no longer valid

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2023-12-12 07:45:12 +00:00
teor 1ccf5fba46
scan(test): Implement scanner format round-trip tests (#8071)
* Implement scanner format round-trip tests

* Limit random heights to database range

* Increase coverage of state height serialization proptests #7443
2023-12-07 19:43:49 +00:00
teor 9fec7116ca
cleanup(db): Give forward and reverse database iterators explicit names (#8063)
* Give forward and reverse database iterators explicit names

* clippy: Ignore large error variants
2023-12-06 21:57:43 +00:00
teor 36f226362d
change(scan): Store one transaction ID per database row, to make queries easier (#8062)
* Upgrade the scanner database major version to 1

* Update format docs

* Change the high-level scanner db format

* Change the scanner serialization formats

* Fix value format and tests

* Fix incorrect types

* Update documentation

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2023-12-06 17:34:21 +00:00
Marek d3dc7d0f0e
change(scan): Store scanned TXIDs in "display order" (#8057)
* Store scanned TXIDs in "display order"

* Unrelated: Remove a redundant `Arc`

---------

Co-authored-by: teor <teor@riseup.net>
2023-12-05 22:49:39 +00:00
teor 4306a00f3c
Scan blocks with sapling keys and write the results to the database (#8040)
* Fix availability of tokio::time in scanner

* Create a function that parses a key into a list of keys

* Pass a ChainTipChange to the scanner function

* Convert a scanned block to a sapling result

* Make it easier to pass keys and blocks

* Increase scanner wait times

* Parse keys once at the start of the scan

* Get a block from the state instead of the tip

* Don't log secret keys, only log every 100,000 blocks

* Scan each block and add the results to storage

* Move blocking tasks into spawn_blocking()

* Update the acceptance test

* Use a dummy sapling tree size

* Use a larger dummy size
2023-12-03 21:58:48 +00:00
teor db05845f98
change(scan): Use the on-disk database for keys and results (#8036)
* Expose IntoDisk and FromDisk in zebra-state

* Implement database serialization for SaplingScanningKey Strings

* Implement serialization for Vec<SaplingScannedResult> (Vec<transaction::Hash>)

* Implement seralization for SaplingScannedDatabaseIndex

* Add an is_empty() method

* Add a read method for a specific index, and document it

* Implement writing scanner results to the database

* Make read name more explicit

* Implement writing scanner keys

* Implement reading sapling keys

* Spawn blocking tasks correctly in async code

* Change storage results methods to use the database

* Update tests that use storage

* Use spawn_blocking() for database methods

* Change the check interval to slightly less than the block interval

* Expose raw database methods with shielded-scan

* fix `scan_task_starts` test

* minor doc change in test

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2023-11-30 21:27:46 +00:00
teor 8c717c92dd
change(scan): Create a scanner storage database, but don't use it yet (#8031)
* Create an empty storage/db module

* Use ephemeral storage in tests

* Populate storage inside new() method

* Move scanner setup into an init() method

* Pass the network to scanner init

* Create a database but don't actually use it

* Skip shutdown format checks when skipping format upgrades

* Allow the scanner to skip launching format upgrades in production

* Refactor skipping format upgrades so it is consistent

* Allow checking configs for equality

* Restore Network import
2023-11-30 12:59:15 +00:00
teor 1d241afbaa
change(state): Expose ZebraDb methods that can create different kinds of databases (#8002)
* Provide access to DiskDb and DiskWriteBatch outside the state using a feature

* Actually let's export ZebraDb for the format upgrade code

* Pass column families to ZebraDb as an argument

* Allow the database kind to be changed in config.rs

* Use the state kind in finalized_state.rs

* Allow different database kinds in ZebraDb, but don't move the upgrade code yet

* Allow different database kinds in DiskDb

* Allow different database kinds in upgrade.rs, but don't split the upgrade code out yet

* Add new arguments to raw database tests

* Fix doc links

* Fix internal imports

* Fix unused code

* Update zebrad version metadata

* Create a specific state database delete function

* Fix state exports

* Fix zebrad tests

* Fix zebrad state write tests

* Make CI run again

* Fix dead code warnings for test methods

* Remove unnecessary async on some tests

* Fix logging required by tests

* Fix logging required in test itself

* Fix variable names

* Try to copy the message and add regexes
2023-11-28 13:49:11 +00:00
teor d20ec3940e
fix(db): Fix a sprout/history tree read panic in Zebra 1.4.0, which only happens before the 25.3.0 state upgrade completes (#7972)
* Avoid a race condition in the 25.3.0 state upgrade

* Look for the most recent sprout tree height if needed

* Get the latest history tree not the tip height history tree

* Discard keys provided by low level database method

* Remove extra ;

* Provide key types & rustfmt

* Fix weird closure type syntax

* Update zebra-state/src/service/finalized_state/disk_format/upgrade/fix_tree_key_type.rs

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2023-11-22 23:38:56 +00:00
teor d689e7344b
fix(net): Fix potential network hangs, and reduce code complexity (#7859)
* Refactor return type of poll_discover()

* Simplify poll_ready() by removing preselected peers

* Fix peer set readiness check

* Pass task context correctly to background tasks

* Make poll_discover() return Pending

* Make poll_inventory() return Pending

* Make poll_unready() return Poll::Pending

* Simplify with futures::ready!() and ?

* When there are no peers, wake on newly ready peers, or new peers, in that order

* Preserve the original waker when there are no unready peers

* Fix polling docs and remove unnecessary code

* Make sure we're ignoring Poll::Pending not Result

* Make panic checking method names clearer

* Fix connection Client task wakeups and error handling

* Cleanup connection panic handling and add wakeup docs

* Fix connection client task wakeups to prevent hangs

* Simplify error and pending handling

* Clarify inventory set behaviour

* Define peer set poll_* methods so they return Ok if they do something

* Clarify documentation

Co-authored-by: Arya <aryasolhi@gmail.com>

* Fix test that depended on preselected peers

* Check ready peers for errors before sending requests to them

* Fix a hanging test by not waiting for irrelevant actions

* Only remove cancel handles when they are required

* fix incorrect panic on termination setting

* Clarify method comments

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-11-16 19:53:24 +00:00
teor af82a76d18
cleanup(clippy): Remove unnecessary `try_into()` with `cargo clippy --fix` (#7940)
* Remove unnecessary try_into() with cargo clippy --fix

* Manually remove some unnecessary try_from()s

* impl From<Diversifier> for pallas::Affine instead of TryFrom

* Remove unused imports

* cargo fmt --all

* Remove redundant comma
2023-11-14 20:31:49 +00:00
Marek 2fad3573fd
change(state): Make the types for finalized blocks consistent (#7923)
* Stop using `SemanticallyVerifiedBlockWithTrees`

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Doc that default treestate corresponds to genesis

* Doc the diff between default values of tree roots

---------

Co-authored-by: teor <teor@riseup.net>
2023-11-08 22:06:16 +00:00
teor afbe807060
change(docs): Replace doc.zebra.zfnd.org API documentation with docs.rs (#7886)
* Simple replacements of doc.zebra.zfnd.org with docs.rs

* Manual fixes for specific main/internal/external docs

* Point developer docs to doc-internal.zebra.zfnd.org

* fastmod --glob '\!.git' -- doc.zebra.zfnd.org/zebrad docs.rs/zebrad/latest/zebrad

* Manually remove any remaining doc.zfnd.zebra.org links

* Remove the external docs job

* Add changelog entry and fix links

* Fix links that were broken before this PR
2023-11-01 22:09:40 +00:00
teor b78c9dd802
change(deps): `cargo update` on commit bd06294 on 2023-11-01 (#7885)
* `cargo update` on commit bd06294 on 2023-11-01

```
    Updating crates.io index
    Updating ahash v0.8.3 -> v0.8.6
    Updating async-trait v0.1.73 -> v0.1.74
    Updating base64 v0.21.4 -> v0.21.5
      Adding bindgen v0.68.1
      Adding const-oid v0.9.5
    Updating cpufeatures v0.2.9 -> v0.2.11
    Updating curve25519-dalek-derive v0.1.0 -> v0.1.1
    Updating cxx-gen v0.7.109 -> v0.7.110
      Adding der v0.7.8
    Updating deranged v0.3.8 -> v0.3.9
    Updating ed25519 v2.2.2 -> v2.2.3
    Updating fiat-crypto v0.2.1 -> v0.2.2
    Updating hashbrown v0.14.1 -> v0.14.2
    Updating hyper-rustls v0.24.1 -> v0.24.2
    Updating iana-time-zone v0.1.57 -> v0.1.58
    Updating indexmap v2.0.2 -> v2.1.0
    Updating ipnet v2.8.0 -> v2.9.0
    Updating mio v0.8.8 -> v0.8.9
    Updating parking_lot_core v0.9.8 -> v0.9.9
    Updating pest v2.7.4 -> v2.7.5
    Updating pest_derive v2.7.4 -> v2.7.5
    Updating pest_generator v2.7.4 -> v2.7.5
    Updating pest_meta v2.7.4 -> v2.7.5
      Adding pkcs8 v0.10.2
    Updating portable-atomic v1.4.3 -> v1.5.1
      Adding powerfmt v0.2.0
    Removing redox_syscall v0.3.5
    Updating rgb v0.8.36 -> v0.8.37
      Adding ring v0.17.5
    Updating rustls v0.21.7 -> v0.21.8
    Updating rustls-webpki v0.101.6 -> v0.101.7
    Updating sct v0.7.0 -> v0.7.1
    Removing socket2 v0.4.9
    Removing socket2 v0.5.4
      Adding socket2 v0.4.10
      Adding socket2 v0.5.5
      Adding spki v0.7.2
    Updating time v0.3.29 -> v0.3.30
    Updating toml v0.8.3 -> v0.8.6
    Updating toml_datetime v0.6.4 -> v0.6.5
    Updating toml_edit v0.20.3 -> v0.20.7
    Updating tracing-log v0.1.3 -> v0.1.4
      Adding untrusted v0.9.0
    Updating uuid v1.4.1 -> v1.5.0
    Removing windows v0.48.0
      Adding windows-core v0.51.1
    Updating winnow v0.5.16 -> v0.5.18
      Adding zerocopy v0.7.21
      Adding zerocopy-derive v0.7.21
```

* Allow duplicate ring dependency in hdwallet

* Remove an unused import warning
2023-11-01 18:26:16 +00:00
Marek 0d3892f61f
change(state): Refactor the naming of note commitment subtrees (#7855)
* Rename `node` & `Node` to `root` & `Root`

* Rename `end` to `end_height`

* Rename `Root` to `SubtreeRoot`
2023-10-30 20:06:54 +00:00
teor df55201c87
chore(clippy): Cleanup nightly clippy warnings from 2023-10-30 (#7868)
* Use first() rather than get(0)

* Remove unused imports and silence test-related warnings

* More clippy fixes
2023-10-30 12:21:05 +00:00