Commit Graph

792 Commits

Author SHA1 Message Date
Arya 15e1096826
change(consensus): Allow configurable NU5 activation height on Regtest (#8505)
* Allow configurable Nu5 activation height on Regtest

* Fixes test

* removes outdated TODO

* Adds `current_with_activation_height()` method and uses it in `Commitment::from_bytes()`
2024-05-08 14:46:13 +00:00
Arya 3e264c73a6
allows square brackets in network config listen addr / external add (#8504) 2024-05-07 19:47:08 +00:00
Alfredo Garcia 239fcc85ba
feat(network): Configurable external address for Version network messages (#8488)
* add configurable external address for Version network messages

* apply suggestions from code review

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2024-05-03 01:07:52 +00:00
Arya 8a786fe6ce
add(consensus): Adds block construction and validation for NU5 from block height 1 for Regtest (#8475)
* Always activate Canopy at Height(1) on Regtest

* Ignores the zip 212 grace period on configured Testnets and Regtest

* - Returns early when there is no Heartwood activation height when creating or updating HistoryTree
- Skips call to `check::legacy_chain()` when no NU5 activation height is set (it would return immediately anyway)
- Replaces `.map()` with `.filter_map()` in `NetworkUpgrade::target_spacings()`
- Removes outdated TODO

* - When proof of work is disabled, skips checking if Zebra is synced to the network tip in the getblocktemplate method

* Sets full_verifier_utxo_lookahead to Height::MIN instead of panicking

* When network is regtest, skips starting sync task and commits the genesis block if it's missing in the state

* updates/fixes test config

* Adds test for committing Canopy blocks on Regtest

* - Updates median time past and difficulty checks to use fewer than 11/17/28 blocks

* uses SLOW_START_INTERVAL of 0 if PoW is disabled, adds TODOs

* Update getblocktemplate method to return reserved chain history activation root hash, uses Nu5 at height 1

Test passes.

* Updates test to expect NU5 as the default nu activation at Height(1)

* Removes invalid difficulty snapshot

* fixes tests

* removes regtest NU5 activation height config field

* Apply suggestions from code review

* Update zebra-state/src/service/check/difficulty.rs

Co-authored-by: Marek <mail@marek.onl>

* Update zebra-state/src/service/check/difficulty.rs

* Updates docs/comments, renames an argument

---------

Co-authored-by: Marek <mail@marek.onl>
2024-05-02 19:43:42 +00:00
Arya e5d4d739b7
add(consensus/network): Adds `new_regtest()` constructors to `testnet::Parameters` and `Network` (#8413)
* minor cleanup and rename

* Adds an empty NetworkParameters struct to Network::Testnet variant, updates production code.

* Updates tests

* Adds `NetworkKind` and uses it instead of `Network` in `HistoryTreeParts` and `transparent::Address`

* Adds a [network.testnet_parameters] section to the config, uses `NetworkKind` as zebra_network::Config::network field type, and converts 'Network' to `NetworkKind` before serializing

* Applies some suggestions from code review

* Applies suggestions from code review

* returns b58 prefix constants directly to remove From<NetworkKind> impl for zcash_primitives::consensus::Network

* Applies more suggestions from code review.

* moves conversions to zcash_primitives::consensus::Network to where they're used.

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

* rename `network` variables and method names typed as NetworkKind to `network_kind`

* use only test block heights for the network associated with them

* Applies more suggestions from code review.

* Rename `NetworkParameters` to `Parameters` and move it a new `testnet` module

* adds activation heights field

* updates stored test config, adds a quicker test for checking that stored configs can be parsed, adds an intermediate representation of activation heights

* implement Parameters for Network

* Passes &Network directly instead of converting to zp_consensus::Network where there were conversions

* fixes a bad merge (removes a network conversion in zcash_note_encryption)

* Adds a test for the Parameters impl for zebra_chain::Network

* fixes doc links

* - Makes the `activation_heights` config field optional by adding a #[serde(default)]
- Panics if a non-zero activation height is provided for the `Genesis` network upgrade
- Always sets the `Genesis` and `BeforeOverwinter` network upgrade activation heights to 0 and 1, `BeforeOverwinter` could be overwritten by a later network upgrade
- Makes the `activation_heights` field on `Parameters` private, adds/uses an accessor method instead, and adds a builder struct and `build()` method

* small refactor of activation_heights() method

* check that activation heights are in the right order

* Updates `NetworkUpgrade::activation_height()` to return the height of the next NetworkUpgrade if it doesn't find the activation height of `&self`

* checks that the miner address is of TestnetKind on Regtest and update assertion message

* Adds a DNetworkUpgradeActivationHeights struct for better control over how activation heights can be configured

* moves all ordered network upgrades to a constant, adds a no_duplicates test, moves struct with activation heights outside deserialization impl and accepts it in `ParametersBuilder::activation_heights()` instead of a Vec

* panics if any network upgrades are configured to activate at Height(0) because it's reserved for Genesis

* Simplifies the `ParametersBuilder::activation_heights()` method and removes an unnecessary test.

* Adds Sapling HRPs as fields on testnet params. (#8398)

* Applies suggestions from code review.

* Update zebra-chain/src/parameters/network_upgrade.rs

Co-authored-by: Marek <mail@marek.onl>

* Adds `network_name` field and accessor method on `Parameters`, uses it in the `Display` impl for `Network`

* Removes unnecessary `.filter()`

* adds constraints on valid network names and a test

* adds config field for setting network name, adds "with_" prefix to ParameterBuilder setter methods

* Adds `MainnetKind`, `TestnetKind`, and `RegtestKind` to reserved network names

* updates stored test configs and fixes `network_name()` docs

* Adds a `new_regtest()` method on `Network` and `testnet::Parameters`, updates config deserialization to return an error if the initial_testnet_peers include any default initial peers AND contain configured activation heights

* Updates `activates_network_upgrades_correctly` test to check Regtest activation heights (and default Mainnet/Testnet)

* Refactors if-let & match statement into general match statement, removes constraint against including `testnet_parameters` field/section in the config when using `Mainnet` or `Regtest`

* Removes outdated TODO

* Restores `testnet_parameters` section of the latest stored config.

* Adds `with_sapling_hrps()` method and uses it to set the Regtest HRPs in `new_regtest()`.

Adds a test for Sapling HRP validation

* Checks that default Mainnet/Testnet/Regtest Sapling HRPs pass validation in `with_sapling_hrps()`

* Uses the correct constant in test

* Adds `is_regtest()` methods

* Updates test docs

* drop custom panic hooks after expected panics

---------

Co-authored-by: Marek <mail@marek.onl>
2024-04-25 04:04:05 +00:00
Arya 275e99ec72
add(chain): Adds a `network_name` field to `testnet::Parameters` (#8411)
* minor cleanup and rename

* Adds an empty NetworkParameters struct to Network::Testnet variant, updates production code.

* Updates tests

* Adds `NetworkKind` and uses it instead of `Network` in `HistoryTreeParts` and `transparent::Address`

* Adds a [network.testnet_parameters] section to the config, uses `NetworkKind` as zebra_network::Config::network field type, and converts 'Network' to `NetworkKind` before serializing

* Applies some suggestions from code review

* Applies suggestions from code review

* returns b58 prefix constants directly to remove From<NetworkKind> impl for zcash_primitives::consensus::Network

* Applies more suggestions from code review.

* moves conversions to zcash_primitives::consensus::Network to where they're used.

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

* rename `network` variables and method names typed as NetworkKind to `network_kind`

* use only test block heights for the network associated with them

* Applies more suggestions from code review.

* Rename `NetworkParameters` to `Parameters` and move it a new `testnet` module

* adds activation heights field

* updates stored test config, adds a quicker test for checking that stored configs can be parsed, adds an intermediate representation of activation heights

* implement Parameters for Network

* Passes &Network directly instead of converting to zp_consensus::Network where there were conversions

* fixes a bad merge (removes a network conversion in zcash_note_encryption)

* Adds a test for the Parameters impl for zebra_chain::Network

* fixes doc links

* - Makes the `activation_heights` config field optional by adding a #[serde(default)]
- Panics if a non-zero activation height is provided for the `Genesis` network upgrade
- Always sets the `Genesis` and `BeforeOverwinter` network upgrade activation heights to 0 and 1, `BeforeOverwinter` could be overwritten by a later network upgrade
- Makes the `activation_heights` field on `Parameters` private, adds/uses an accessor method instead, and adds a builder struct and `build()` method

* small refactor of activation_heights() method

* check that activation heights are in the right order

* Updates `NetworkUpgrade::activation_height()` to return the height of the next NetworkUpgrade if it doesn't find the activation height of `&self`

* checks that the miner address is of TestnetKind on Regtest and update assertion message

* Adds a DNetworkUpgradeActivationHeights struct for better control over how activation heights can be configured

* moves all ordered network upgrades to a constant, adds a no_duplicates test, moves struct with activation heights outside deserialization impl and accepts it in `ParametersBuilder::activation_heights()` instead of a Vec

* panics if any network upgrades are configured to activate at Height(0) because it's reserved for Genesis

* Simplifies the `ParametersBuilder::activation_heights()` method and removes an unnecessary test.

* Adds Sapling HRPs as fields on testnet params. (#8398)

* Applies suggestions from code review.

* Update zebra-chain/src/parameters/network_upgrade.rs

Co-authored-by: Marek <mail@marek.onl>

* Adds `network_name` field and accessor method on `Parameters`, uses it in the `Display` impl for `Network`

* Removes unnecessary `.filter()`

* adds constraints on valid network names and a test

* adds config field for setting network name, adds "with_" prefix to ParameterBuilder setter methods

* Adds `MainnetKind`, `TestnetKind`, and `RegtestKind` to reserved network names

* updates stored test configs and fixes `network_name()` docs

---------

Co-authored-by: Marek <mail@marek.onl>
2024-04-24 11:03:37 +00:00
Arya f8c153991b
add(consensus): Add `activation_heights` field to `NetworkParameters` and implement `Parameters` for `Network` (#8383)
* minor cleanup and rename

* Adds an empty NetworkParameters struct to Network::Testnet variant, updates production code.

* Updates tests

* Adds `NetworkKind` and uses it instead of `Network` in `HistoryTreeParts` and `transparent::Address`

* Adds a [network.testnet_parameters] section to the config, uses `NetworkKind` as zebra_network::Config::network field type, and converts 'Network' to `NetworkKind` before serializing

* Applies some suggestions from code review

* Applies suggestions from code review

* returns b58 prefix constants directly to remove From<NetworkKind> impl for zcash_primitives::consensus::Network

* Applies more suggestions from code review.

* moves conversions to zcash_primitives::consensus::Network to where they're used.

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

* rename `network` variables and method names typed as NetworkKind to `network_kind`

* use only test block heights for the network associated with them

* Applies more suggestions from code review.

* Rename `NetworkParameters` to `Parameters` and move it a new `testnet` module

* adds activation heights field

* updates stored test config, adds a quicker test for checking that stored configs can be parsed, adds an intermediate representation of activation heights

* implement Parameters for Network

* Passes &Network directly instead of converting to zp_consensus::Network where there were conversions

* fixes a bad merge (removes a network conversion in zcash_note_encryption)

* Adds a test for the Parameters impl for zebra_chain::Network

* fixes doc links

* - Makes the `activation_heights` config field optional by adding a #[serde(default)]
- Panics if a non-zero activation height is provided for the `Genesis` network upgrade
- Always sets the `Genesis` and `BeforeOverwinter` network upgrade activation heights to 0 and 1, `BeforeOverwinter` could be overwritten by a later network upgrade
- Makes the `activation_heights` field on `Parameters` private, adds/uses an accessor method instead, and adds a builder struct and `build()` method

* small refactor of activation_heights() method

* check that activation heights are in the right order

* Updates `NetworkUpgrade::activation_height()` to return the height of the next NetworkUpgrade if it doesn't find the activation height of `&self`

* checks that the miner address is of TestnetKind on Regtest and update assertion message

* Adds a DNetworkUpgradeActivationHeights struct for better control over how activation heights can be configured

* moves all ordered network upgrades to a constant, adds a no_duplicates test, moves struct with activation heights outside deserialization impl and accepts it in `ParametersBuilder::activation_heights()` instead of a Vec

* panics if any network upgrades are configured to activate at Height(0) because it's reserved for Genesis

* Simplifies the `ParametersBuilder::activation_heights()` method and removes an unnecessary test.

* Adds Sapling HRPs as fields on testnet params. (#8398)

* Applies suggestions from code review.

* Update zebra-chain/src/parameters/network_upgrade.rs

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2024-04-19 01:20:34 +00:00
Alfredo Garcia 887d4a2457
fix(beta-toolchain): Warnings (#8404)
* fix beta imports and other warnings

* clippy lints beta

* move attributes to after docs

Co-authored-by: Marek <mail@marek.onl>

* move some code to test module

* implement display instead of direct tostring for LongPollId

Co-authored-by: Arya <aryasolhi@gmail.com>

* fix typo

---------

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
2024-04-18 01:45:22 +00:00
Arya 16a39f89d0
add(consensus/network): Add an empty `Parameters` struct in `Network::Testnet` (#8368)
* minor cleanup and rename

* Adds an empty NetworkParameters struct to Network::Testnet variant, updates production code.

* Updates tests

* Adds `NetworkKind` and uses it instead of `Network` in `HistoryTreeParts` and `transparent::Address`

* Adds a [network.testnet_parameters] section to the config, uses `NetworkKind` as zebra_network::Config::network field type, and converts 'Network' to `NetworkKind` before serializing

* Applies some suggestions from code review

* Applies suggestions from code review

* returns b58 prefix constants directly to remove From<NetworkKind> impl for zcash_primitives::consensus::Network

* Applies more suggestions from code review.

* moves conversions to zcash_primitives::consensus::Network to where they're used.

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

* rename `network` variables and method names typed as NetworkKind to `network_kind`

* use only test block heights for the network associated with them

* Applies more suggestions from code review.

* Rename `NetworkParameters` to `Parameters` and move it a new `testnet` module

---------

Co-authored-by: Marek <mail@marek.onl>
2024-04-17 02:20:34 +00:00
idky137 4579722833
change(chain): Remove `Copy` trait impl from `Network` (#8354)
* removed derive copy from network, address and ledgerstate

* changed is_max_block_time_enforced to accept ref

* changed NetworkUpgrade::Current to accept ref

* changed NetworkUpgrade::Next to accept ref

* changed NetworkUpgrade::IsActivationHeight to accept ref

* changed NetworkUpgrade::TargetSpacingForHeight to accept ref

* changed NetworkUpgrade::TargetSpacings to accept ref

* changed NetworkUpgrade::MinimumDifficultySpacing_forHeight to accept ref

* changed NetworkUpgrade::IsTestnetMinDifficultyBlock to accept ref

* changed NetworkUpgrade::AveragingWindowTimespanForHeight to accept ref

* changed NetworkUpgrade::ActivationHeight to accept ref

* changed sapling_activation_height to accept ref

* fixed lifetime for target_spacings

* fixed sapling_activation_height

* changed transaction_to_fake_v5 and fake_v5_transactions_for_network to accept ref to network

* changed Input::vec_strategy to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_history.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/viewing_key* to accept ref to network

* changed functions in zebra-chain/src/transparent/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_primitives.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_note_encryption.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/history_tree* to accept ref to network

* changed functions in zebra-chain/src/block* to accept ref to network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* fixed errors in zebra-chain/src/block/arbitrary.rs

* finished fixing errors in zebra-chain - all crate tests pass

* changed functions in zebra-state::service::finalized_state to accept &Network

* changed functions in zebra-state::service::non_finalized_state to accept &Network

* zebra-state tests run but fail with overflow error

* zebra-state tests all pass

* converted zebra-network -- all crate tests pass

* applied all requested changes from review

* converted zebra-consensus -- all crate tests pass

* converted zebra-scan -- all crate tests pass

* converted zebra-rpc -- all crate tests pass

* converted zebra-grpc -- all crate tests pass

* converted zebrad -- all crate tests pass

* applied all requested changes from review

* fixed all clippy errors

* fixed build error in zebrad/src/components/mempool/crawler.rs
2024-03-19 20:45:27 +00:00
Za Wilcox 3bef54b764
change(consensus): Refactor production code for network consensus rules to `Network` methods (#8340)
* begin refactor suggested as "step 2": https://github.com/ZcashFoundation/zebra/issues/7968#issue-2003245309
Squashed from multiple commits to enable partial rebase

* break out more little traits

* add activation implementation leveraging From<Network> for lrz::cons::

* for transfer of ownership I cannot return a type that's owned by the method

* hrp_sapling_extended_full_viewing_key

* complete implementation of interface of Parameters on Network reuse Parameters on zcash Network where possible

* move doc-comments to trait declarations (from impls)

* Simplify/complete Parameters impl for Network

* Add checkpoint_list method, move documentation, etc

* move last match network to inside network method

* add back comment

* use zcash_address for parameter types in zebra-chain

* use inherent methods instead of big parameters passthrough

* revert to implementation of From on zcash_primitives::..::Network vs &zcash_prim...

* move match

* add test to block maximum time rule

* update changelog

* finish porting target_difficutly_limit

* remove obscelete code comment

* revert non-functional change

* finish migrating target_difficulty_limit, checkpoint_list

* update changelog

---------

Co-authored-by: Hazel OHearn <gygaxis@zingolabs.org>
2024-03-12 21:41:44 +00:00
dependabot[bot] f21d7c6934
build(deps): bump the prod group with 6 updates (#8125)
* build(deps): bump the prod group with 6 updates

Bumps the prod group with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [futures](https://github.com/rust-lang/futures-rs) | `0.3.29` | `0.3.30` |
| [tokio](https://github.com/tokio-rs/tokio) | `1.35.0` | `1.35.1` |
| [metrics](https://github.com/metrics-rs/metrics) | `0.21.1` | `0.22.0` |
| [metrics-exporter-prometheus](https://github.com/metrics-rs/metrics) | `0.12.2` | `0.13.0` |
| [reqwest](https://github.com/seanmonstar/reqwest) | `0.11.22` | `0.11.23` |
| [owo-colors](https://github.com/jam1garner/owo-colors) | `3.5.0` | `4.0.0` |


Updates `futures` from 0.3.29 to 0.3.30
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.29...0.3.30)

Updates `tokio` from 1.35.0 to 1.35.1
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.35.0...tokio-1.35.1)

Updates `metrics` from 0.21.1 to 0.22.0
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/compare/metrics-v0.21.1...metrics-v0.22.0)

Updates `metrics-exporter-prometheus` from 0.12.2 to 0.13.0
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/compare/metrics-exporter-prometheus-v0.12.2...metrics-exporter-prometheus-v0.13.0)

Updates `reqwest` from 0.11.22 to 0.11.23
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.22...v0.11.23)

Updates `owo-colors` from 3.5.0 to 4.0.0
- [Commits](https://github.com/jam1garner/owo-colors/compare/v3.5.0...v4.0.0)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: metrics
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: metrics-exporter-prometheus
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: owo-colors
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>

* update all metric macros

* fix deprecated function

* fix duplicated deps

* Fix an incorrect gauge method call

* Expand documentation and error messages for best chain length

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2024-01-02 01:26:54 +00:00
teor ad015e04d9
Move history tree and value balance to typed column families (#8115)
* impl TryFrom<zcash_primitives::BlockHeight> for Height

* Add type-safe read and write database methods

* Only allow typed access to the scanner DB

* Update docs

* Implement a common method as a trait

* Fix imports

* Tidy state imports

* Activate tracing logging macros in the whole scanner crate

* Fix dead code warnings

* Use a more sensible export order

* Remove a 1.72 lint exception now 1.74 is stable

* Switch history trees over to TypedColumnFamily, and remove redundant code

* Add typed batch creation methods, and switch history trees to them

* Convert ValueBalance to typed column families

* Make the APIs compatible after a merge

* Use `ZebraDb` instead of `DiskDb` where needed

---------

Co-authored-by: Marek <mail@marek.onl>
2023-12-20 23:20:57 +00:00
teor 6306a755de
change(log): Silence verbose failed connection logs (#8072)
* Silence verbose failed connection logs

* Update comment and fix formatting
2023-12-07 03:34:52 +00:00
teor 5d80c646fe
clippy: fix implied iterator bounds (#8052) 2023-12-05 04:06:42 +00:00
teor 8c717c92dd
change(scan): Create a scanner storage database, but don't use it yet (#8031)
* Create an empty storage/db module

* Use ephemeral storage in tests

* Populate storage inside new() method

* Move scanner setup into an init() method

* Pass the network to scanner init

* Create a database but don't actually use it

* Skip shutdown format checks when skipping format upgrades

* Allow the scanner to skip launching format upgrades in production

* Refactor skipping format upgrades so it is consistent

* Allow checking configs for equality

* Restore Network import
2023-11-30 12:59:15 +00:00
teor 4aaeb4c4cc
cleanup(net): Deleted unused NeverAttemptedAlternate peer state (#7982)
* Update Connection debug impl missed in previous PRs

* Add an initial_cached_addrs argument to Connection::new()

* Stop sending version and remote IP peers directly to the address book

* Remove the NeverAttemptedAlternate peer state

* Remove the NewAlternate MetaAddrChange

* Remove the never_attempted_alternate AddressBookMetrics

* Update CandidateSet docs and add a Debug impl
2023-11-28 00:30:13 +00:00
teor f08cc2dc50
security(net): Stop sending peer addresses from handshakes directly to the address book (#7977)
* Update Connection debug impl missed in previous PRs

* Add an initial_cached_addrs argument to Connection::new()

* Stop sending version and remote IP peers directly to the address book

* Update zebra-network/src/peer/connection.rs

Co-authored-by: teor <teor@riseup.net>

---------

Co-authored-by: Marek <mail@marek.onl>
2023-11-24 01:21:56 +00:00
teor ca062d04d4
Document how outbound handshakes are rate-limited (#7976) 2023-11-23 21:58:59 +00:00
Arya 5e4c0f973f
security: Rate limit GetAddr responses (#7955)
* Updates ADDR_RESPONSE_LIMIT_DENOMINATOR to 4

* Moves logic getting a fraction of Zebra's peers to a method in the address book

* Adds and uses CachedPeerAddrs struct in inbound service

* moves and documents constant

* fixes test

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* updates docs

* renames sanitized_window method

* renames CachedPeerAddrs to CachedPeerAddrResponse

* updates test

* moves try_refresh to per request

* Make unused sanitization method pub(crate)

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* moves CachedPeerAddrResponse to a module

* updates unit test

* fixes unit test

* removes unnecessary condition

* clears cached getaddr response if it can't refresh for over a minute after the refresh time

* tests that inbound service gives out the same addresses for every Peers request before the refresh interval

* Applies suggestion from code review

* fixes doc link

* renames constant

* Fix docs on new constant

* applies suggestion from code review

* uses longer cache expiry time

* Adds code comments

---------

Co-authored-by: teor <teor@riseup.net>
2023-11-21 03:32:23 +00:00
teor 3be22b2cda
security: Limit how many addresses we use from each peer address message (#7952)
* Reformat a method to make related code clearer

* Limit how many addresses we take from each peer response

* Only put extra addresses back in the cache if there are actually any extras

* Refactor address caching into a single method

* Use the same method for updating the peer cache and taking peers from it

* Choose response peers at random, and partially randomly drop excess peers

* Update docs for this fix

* Fix an index bug and clarify a comment

* Add a proptest for cache updates
2023-11-19 21:43:09 +00:00
teor d689e7344b
fix(net): Fix potential network hangs, and reduce code complexity (#7859)
* Refactor return type of poll_discover()

* Simplify poll_ready() by removing preselected peers

* Fix peer set readiness check

* Pass task context correctly to background tasks

* Make poll_discover() return Pending

* Make poll_inventory() return Pending

* Make poll_unready() return Poll::Pending

* Simplify with futures::ready!() and ?

* When there are no peers, wake on newly ready peers, or new peers, in that order

* Preserve the original waker when there are no unready peers

* Fix polling docs and remove unnecessary code

* Make sure we're ignoring Poll::Pending not Result

* Make panic checking method names clearer

* Fix connection Client task wakeups and error handling

* Cleanup connection panic handling and add wakeup docs

* Fix connection client task wakeups to prevent hangs

* Simplify error and pending handling

* Clarify inventory set behaviour

* Define peer set poll_* methods so they return Ok if they do something

* Clarify documentation

Co-authored-by: Arya <aryasolhi@gmail.com>

* Fix test that depended on preselected peers

* Check ready peers for errors before sending requests to them

* Fix a hanging test by not waiting for irrelevant actions

* Only remove cancel handles when they are required

* fix incorrect panic on termination setting

* Clarify method comments

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-11-16 19:53:24 +00:00
teor af82a76d18
cleanup(clippy): Remove unnecessary `try_into()` with `cargo clippy --fix` (#7940)
* Remove unnecessary try_into() with cargo clippy --fix

* Manually remove some unnecessary try_from()s

* impl From<Diversifier> for pallas::Affine instead of TryFrom

* Remove unused imports

* cargo fmt --all

* Remove redundant comma
2023-11-14 20:31:49 +00:00
teor 00f00d9840
cleanup(net): Delete duplicate negotiated version code (#7912)
* Delete duplicate negotiated version code

* Create ConnectionInfo earlier to simplify return value
2023-11-07 04:02:44 +00:00
teor 43e54d1cb2
fix(net): Fix a potential hang caused by accessing the address book directly (#7902)
* Fix a potential hang accessing the address book directly

* Remove unused connection shutdown MetaAddr arguments

* Add an UpdateConnected MetaAddrChange, that sends initial connection info

* Fix some tests

* Fix a panic with a zero channel size
2023-11-05 22:28:58 +00:00
teor 628b3e39af
fix(net): Add outer timeouts for critical network operations to avoid hangs (#7869)
* Refactor out try_to_sync_once()

* Add outer timeouts for obtaining and extending tips

* Refactor out request_genesis_once()

* Wrap genesis download once in a timeout

* Increase the genesis timeout to avoid denial of service from old nodes

* Add an outer timeout to mempool crawls

* Add an outer timeout to mempool download/verify

* Remove threaded mutex blocking from the inbound service

* Explain why inbound readiness never hangs

* Fix whitespace that cargo fmt doesn't

* Avoid hangs by always resetting the past lookahead limit flag

* Document block-specific and syncer-wide errors

* Update zebrad/src/components/sync.rs

Co-authored-by: Marek <mail@marek.onl>

* Use correct condition for log messages

Co-authored-by: Marek <mail@marek.onl>

* Keep lookahead reset metric

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: Marek <mail@marek.onl>
2023-11-02 15:00:18 +00:00
teor afbe807060
change(docs): Replace doc.zebra.zfnd.org API documentation with docs.rs (#7886)
* Simple replacements of doc.zebra.zfnd.org with docs.rs

* Manual fixes for specific main/internal/external docs

* Point developer docs to doc-internal.zebra.zfnd.org

* fastmod --glob '\!.git' -- doc.zebra.zfnd.org/zebrad docs.rs/zebrad/latest/zebrad

* Manually remove any remaining doc.zfnd.zebra.org links

* Remove the external docs job

* Add changelog entry and fix links

* Fix links that were broken before this PR
2023-11-01 22:09:40 +00:00
teor 7e7f989545
Fix new nightly clippy and rustc lints (#7860) 2023-10-27 19:35:57 +00:00
Alfredo Garcia e4c3f95d27
chore: fix typos (#7862)
Co-authored-by: vuittont60 <81072379+vuittont60@users.noreply.github.com>
2023-10-27 17:45:43 +00:00
Arya 5367ccbc5c
fix(network): Reconnect with peers after brief network interruption (#7853)
* Fixes bug where Zebra won't reconnect to peers after brief loss of network connectivity

* only dial on timercrawl when theres a new address or zero active outbound conns
2023-10-27 06:13:16 +00:00
teor 920ee14512
fix(doc): Fix broken links to `zebra_network` and `zebra_state` `Config` structs on doc.zebra.zfnd.org (#7838)
* Use full module paths to avoid a rustdoc bug

* Exclude zebra-test from external docs
2023-10-27 01:05:26 +00:00
teor ca8d529a09
cleanup(clippy): Fix new nightly clippy lints from July and August 2023 (#7384)
* cargo +nightly clippy --fix --all-features --all-targets

* Manually fix clippy::redundant_locals

* Remove unused deny.toml duplicate dependency exception

* Manually fix clippy::needless_pass_by_ref_mut

* Manually fix -W elided-lifetimes-in-associated-constant

* Manually fix clippy::unnecessary_mut_passed

* Manually fix -W unused_mut

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2023-08-25 22:58:30 +00:00
Alfredo Garcia 000c7b4bc6
fix(clippy): Fix clippy and rustfmt after last releases today (#7381)
* fix clippy and rustfmt after last releases today

* Ignore clippy::arc_with_non_send_sync false positives in proptest code only

* Remove .cargo/config.toml changes that didn't work

---------

Co-authored-by: teor <teor@riseup.net>
2023-08-25 04:08:13 +00:00
Alfredo Garcia 0cc48a322a
fix(docs): docs after new rust version (#7375)
* fix docs build

* fix docs build errors in sapling trees

* fix docs build in sprout joinsplits

* fix doc build in handshake

* fix docs build in zebra-state

* fix docs build in zebrad

* new line fix
2023-08-24 11:31:10 +00:00
teor 3bbe3cec4f
fix(panic): Stop panicking on async task cancellation on shutdown in network and state futures (#7219)
* Add an async-error feature and an initial module structure

* Implement checking for panics in OS threads and async tasks

* Implement waiting for panics in OS threads and async tasks

* Add a TODO to simplify some state request error handling

* Use the new panic-checking methods in zebra-state

* Use new panic-checking methods in zebra-network

* fixup! Implement waiting for panics in OS threads and async tasks

* Replace existing async code with generic panic-checking methods

* Simplify trait to a single method

* Move thread panic code into generic trait impls

* Simplify option handling

Co-authored-by: Arya <aryasolhi@gmail.com>

* Fix comment

Co-authored-by: Arya <aryasolhi@gmail.com>

* Add missing track_caller

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-07-18 04:53:26 +00:00
Arya 7b0dedd3a2
fix(network): Rate-limit inbound connections per IP. (#7041)
* Adds RecentByIp

* Removes new [cfg(not(test))]s, supports configurable max_conn_per_ip in RecentByIp and account_inbound_connections

Updates tests

* Uses self.time_limit instead of constant

* Adds sleep after dropping connections

Uses partition_point & split_off

Moves tests to separate module

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Always prune before adding

* Tweak comments

* Move the time calculation outside the binary search closure

---------

Co-authored-by: teor <teor@riseup.net>
2023-07-14 02:26:46 +00:00
teor f6afec2be8
fix(ci): Increase peer cache startup wait time and test time (#7169)
* Increase peer cache test time

* Wait longer for DNS responses before writing peers to disk
2023-07-07 03:35:41 +00:00
Arya 77ad91ced4
fix(network): Avoid initiating outbound handshakes with IPs for which Zebra already has an active peer. (#7029)
* Adds most_recent_by_ip field to address book

* adds test

* Apply suggestions from code review

* fixes lint

* Updates most_recent_by_ip in .take()

Updates should_update_most_recent_by_ip() and has_active_peer_with_ip to check last_attempt and last_failure times

Renames has_active_peer_with_ip

* Documents that Zebra will not initiate more than 1 outbound connections per IP

* Fixes is_ready_for_connection_attempt_with_ip

Adds test coverage for AttemptPending and Failed

Fixes new_outbound_peer_connections_are_rate_limited proptest

* Applies suggestions from code review.

* Applies suggestions from code review

* Always return true from `is_ready_for_connection_attempt_with_ip` if max_connections_per_ip != 0

* Update max_connections_per_ip config docs

* Warn about invalid config fields and use default values

* Ignores last_attempt and last_failure in is_ready_for_connection_attempt_with_ip

updates test

* Only update most_recent_by_ip if update.last_conn_state is responded.

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* fixes lint

* Update zebra-network/src/address_book.rs

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Fix Rust syntax

* Fix whitespace

---------

Co-authored-by: teor <teor@riseup.net>
2023-07-06 05:54:10 +00:00
teor 147b8fa3a8
cleanup(rust): Fix new nightly clippy warnings (#7135)
* Fix "comparison is always true" warning

* Add missing Send bound

* cargo clippy --fix --all-features --all-targets

* incorrect implementation of clone on a Copy type

* cargo fmt --all
2023-07-05 07:11:27 +00:00
teor f2a2a403a8
fix(log): Remove redundant startup logs, fix progress bar number, order, and wording (#7087)
* Remove duplicate "running" message, send log file opening logs to stderr

* Add missing block progress bar desc, make them consistent

* Put progress bars in a specific order in each section

* Actually make block progress bar desc consistent

* Make progress bar order even more consistent

* Fix fork blocks plural for 1 block

* Use the correct number of chain fork bars

* Disable confusing partial work display

* Add struct field category comments

* Silence a verbose inventory log
2023-07-05 07:08:59 +00:00
teor e6c3b87872
Stop panicking on shutdown in the syncer and network init (#7104) 2023-07-02 20:08:11 +00:00
teor f8e26347a0
Log a zebra-network task cancel on shutdown, rather than panicking (#7078) 2023-06-27 19:42:04 +00:00
teor 3d2c5ef290
fix(concurrency): Use Arc::into_inner() to avoid potential concurrency issues, needs Rust 1.70 (#7032)
* Use Arc::into_inner() to avoid potential concurrency issues

* Remove some outdated clippy lint workarounds (fixed in Rust 1.66)

* Update the required Rust version to 1.70
2023-06-21 20:44:53 +00:00
Arya b40fc9b032
change(network): Configurable maximum connections per IP (#7013)
* Adds config field

* adds new generated config

* Lint

* fixes config_tests
2023-06-20 05:11:45 +00:00
teor 231f5be403
fix(net): Reduce maximum number of connections per IP (#6993)
* Reduce maximum number of connections per IP

* Fix tests that require multiple connections per IP
2023-06-19 18:17:59 +00:00
teor ad7af3e2d8
fix(net): Clean up licensing, closure `move`, log typos, tracing spans (#6995)
* Remove a redundant outbound connector timeout

* Fix panics in inbound connection handshaker

* Refactor to simplify FuturesUnordered types

* Make licensing notes consistent

* Delete redundant `move` in closures

* Fix a log typo

* Add some missing tracing spans
2023-06-19 18:17:39 +00:00
teor 859353b417
fix(panic): Stop panicking when handling inbound connection handshakes (#6984)
* Remove a redundant outbound connector timeout

* Fix panics in inbound connection handshaker

* Refactor to simplify FuturesUnordered types
2023-06-19 03:39:59 +00:00
Arya 73ce8fbef0
fix(network): Limit number of peer connections per IP address, Ignore new peer connections from the same IP and port (#6980)
* Limits num peer conns per ip

* Update zebra-network/src/peer_set/set.rs

* Update zebra-network/src/constants.rs

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Keep old peer connections, rather than replacing them with new connections

* Adds max_conns_per_ip field

Configures the max to usize::MAX for some tests.

* Adds a test to check that max_conns_per_ip is enforced

---------

Co-authored-by: teor <teor@riseup.net>
2023-06-19 00:53:00 +00:00
teor 1e12a58b5f
fix(handshake): Add extra timeout logging to peer TCP connections (#6969)
* Add a missing timeout to outbound TCP connections

* Move inbound handshakes into their own function, replacing the manual span

* Delete a useless manual span in zebra_network::config

* Add an extra timeout to the spawned inbound handshake task
2023-06-15 21:11:24 +00:00
teor 32ea511a73
fix(net): Reduce inbound service overloads and add a timeout (#6950)
* Increase concurrency limit, reduce peer broadcast

* Fix a div_ceil() TODO

* Document security requirements of inbound peer overload handling

* Reduce drop probability and fix its formatting

* Put a 5 second timeout on inbound service requests

* Update drop probability tests

* Add error types, metrics, and logging for InboundTimeout errors
2023-06-15 00:43:41 +00:00