Commit Graph

114 Commits

Author SHA1 Message Date
idky137 4579722833
change(chain): Remove `Copy` trait impl from `Network` (#8354)
* removed derive copy from network, address and ledgerstate

* changed is_max_block_time_enforced to accept ref

* changed NetworkUpgrade::Current to accept ref

* changed NetworkUpgrade::Next to accept ref

* changed NetworkUpgrade::IsActivationHeight to accept ref

* changed NetworkUpgrade::TargetSpacingForHeight to accept ref

* changed NetworkUpgrade::TargetSpacings to accept ref

* changed NetworkUpgrade::MinimumDifficultySpacing_forHeight to accept ref

* changed NetworkUpgrade::IsTestnetMinDifficultyBlock to accept ref

* changed NetworkUpgrade::AveragingWindowTimespanForHeight to accept ref

* changed NetworkUpgrade::ActivationHeight to accept ref

* changed sapling_activation_height to accept ref

* fixed lifetime for target_spacings

* fixed sapling_activation_height

* changed transaction_to_fake_v5 and fake_v5_transactions_for_network to accept ref to network

* changed Input::vec_strategy to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_history.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/viewing_key* to accept ref to network

* changed functions in zebra-chain/src/transparent/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_primitives.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_note_encryption.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/history_tree* to accept ref to network

* changed functions in zebra-chain/src/block* to accept ref to network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* fixed errors in zebra-chain/src/block/arbitrary.rs

* finished fixing errors in zebra-chain - all crate tests pass

* changed functions in zebra-state::service::finalized_state to accept &Network

* changed functions in zebra-state::service::non_finalized_state to accept &Network

* zebra-state tests run but fail with overflow error

* zebra-state tests all pass

* converted zebra-network -- all crate tests pass

* applied all requested changes from review

* converted zebra-consensus -- all crate tests pass

* converted zebra-scan -- all crate tests pass

* converted zebra-rpc -- all crate tests pass

* converted zebra-grpc -- all crate tests pass

* converted zebrad -- all crate tests pass

* applied all requested changes from review

* fixed all clippy errors

* fixed build error in zebrad/src/components/mempool/crawler.rs
2024-03-19 20:45:27 +00:00
dependabot[bot] f21d7c6934
build(deps): bump the prod group with 6 updates (#8125)
* build(deps): bump the prod group with 6 updates

Bumps the prod group with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [futures](https://github.com/rust-lang/futures-rs) | `0.3.29` | `0.3.30` |
| [tokio](https://github.com/tokio-rs/tokio) | `1.35.0` | `1.35.1` |
| [metrics](https://github.com/metrics-rs/metrics) | `0.21.1` | `0.22.0` |
| [metrics-exporter-prometheus](https://github.com/metrics-rs/metrics) | `0.12.2` | `0.13.0` |
| [reqwest](https://github.com/seanmonstar/reqwest) | `0.11.22` | `0.11.23` |
| [owo-colors](https://github.com/jam1garner/owo-colors) | `3.5.0` | `4.0.0` |


Updates `futures` from 0.3.29 to 0.3.30
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.29...0.3.30)

Updates `tokio` from 1.35.0 to 1.35.1
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.35.0...tokio-1.35.1)

Updates `metrics` from 0.21.1 to 0.22.0
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/compare/metrics-v0.21.1...metrics-v0.22.0)

Updates `metrics-exporter-prometheus` from 0.12.2 to 0.13.0
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/compare/metrics-exporter-prometheus-v0.12.2...metrics-exporter-prometheus-v0.13.0)

Updates `reqwest` from 0.11.22 to 0.11.23
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.22...v0.11.23)

Updates `owo-colors` from 3.5.0 to 4.0.0
- [Commits](https://github.com/jam1garner/owo-colors/compare/v3.5.0...v4.0.0)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: metrics
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: metrics-exporter-prometheus
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: owo-colors
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>

* update all metric macros

* fix deprecated function

* fix duplicated deps

* Fix an incorrect gauge method call

* Expand documentation and error messages for best chain length

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2024-01-02 01:26:54 +00:00
teor 6306a755de
change(log): Silence verbose failed connection logs (#8072)
* Silence verbose failed connection logs

* Update comment and fix formatting
2023-12-07 03:34:52 +00:00
teor ca062d04d4
Document how outbound handshakes are rate-limited (#7976) 2023-11-23 21:58:59 +00:00
teor 43e54d1cb2
fix(net): Fix a potential hang caused by accessing the address book directly (#7902)
* Fix a potential hang accessing the address book directly

* Remove unused connection shutdown MetaAddr arguments

* Add an UpdateConnected MetaAddrChange, that sends initial connection info

* Fix some tests

* Fix a panic with a zero channel size
2023-11-05 22:28:58 +00:00
Arya 5367ccbc5c
fix(network): Reconnect with peers after brief network interruption (#7853)
* Fixes bug where Zebra won't reconnect to peers after brief loss of network connectivity

* only dial on timercrawl when theres a new address or zero active outbound conns
2023-10-27 06:13:16 +00:00
teor 3bbe3cec4f
fix(panic): Stop panicking on async task cancellation on shutdown in network and state futures (#7219)
* Add an async-error feature and an initial module structure

* Implement checking for panics in OS threads and async tasks

* Implement waiting for panics in OS threads and async tasks

* Add a TODO to simplify some state request error handling

* Use the new panic-checking methods in zebra-state

* Use new panic-checking methods in zebra-network

* fixup! Implement waiting for panics in OS threads and async tasks

* Replace existing async code with generic panic-checking methods

* Simplify trait to a single method

* Move thread panic code into generic trait impls

* Simplify option handling

Co-authored-by: Arya <aryasolhi@gmail.com>

* Fix comment

Co-authored-by: Arya <aryasolhi@gmail.com>

* Add missing track_caller

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-07-18 04:53:26 +00:00
Arya 7b0dedd3a2
fix(network): Rate-limit inbound connections per IP. (#7041)
* Adds RecentByIp

* Removes new [cfg(not(test))]s, supports configurable max_conn_per_ip in RecentByIp and account_inbound_connections

Updates tests

* Uses self.time_limit instead of constant

* Adds sleep after dropping connections

Uses partition_point & split_off

Moves tests to separate module

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Always prune before adding

* Tweak comments

* Move the time calculation outside the binary search closure

---------

Co-authored-by: teor <teor@riseup.net>
2023-07-14 02:26:46 +00:00
Arya 77ad91ced4
fix(network): Avoid initiating outbound handshakes with IPs for which Zebra already has an active peer. (#7029)
* Adds most_recent_by_ip field to address book

* adds test

* Apply suggestions from code review

* fixes lint

* Updates most_recent_by_ip in .take()

Updates should_update_most_recent_by_ip() and has_active_peer_with_ip to check last_attempt and last_failure times

Renames has_active_peer_with_ip

* Documents that Zebra will not initiate more than 1 outbound connections per IP

* Fixes is_ready_for_connection_attempt_with_ip

Adds test coverage for AttemptPending and Failed

Fixes new_outbound_peer_connections_are_rate_limited proptest

* Applies suggestions from code review.

* Applies suggestions from code review

* Always return true from `is_ready_for_connection_attempt_with_ip` if max_connections_per_ip != 0

* Update max_connections_per_ip config docs

* Warn about invalid config fields and use default values

* Ignores last_attempt and last_failure in is_ready_for_connection_attempt_with_ip

updates test

* Only update most_recent_by_ip if update.last_conn_state is responded.

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* fixes lint

* Update zebra-network/src/address_book.rs

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Fix Rust syntax

* Fix whitespace

---------

Co-authored-by: teor <teor@riseup.net>
2023-07-06 05:54:10 +00:00
teor e6c3b87872
Stop panicking on shutdown in the syncer and network init (#7104) 2023-07-02 20:08:11 +00:00
teor f8e26347a0
Log a zebra-network task cancel on shutdown, rather than panicking (#7078) 2023-06-27 19:42:04 +00:00
teor ad7af3e2d8
fix(net): Clean up licensing, closure `move`, log typos, tracing spans (#6995)
* Remove a redundant outbound connector timeout

* Fix panics in inbound connection handshaker

* Refactor to simplify FuturesUnordered types

* Make licensing notes consistent

* Delete redundant `move` in closures

* Fix a log typo

* Add some missing tracing spans
2023-06-19 18:17:39 +00:00
teor 859353b417
fix(panic): Stop panicking when handling inbound connection handshakes (#6984)
* Remove a redundant outbound connector timeout

* Fix panics in inbound connection handshaker

* Refactor to simplify FuturesUnordered types
2023-06-19 03:39:59 +00:00
Arya 73ce8fbef0
fix(network): Limit number of peer connections per IP address, Ignore new peer connections from the same IP and port (#6980)
* Limits num peer conns per ip

* Update zebra-network/src/peer_set/set.rs

* Update zebra-network/src/constants.rs

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Keep old peer connections, rather than replacing them with new connections

* Adds max_conns_per_ip field

Configures the max to usize::MAX for some tests.

* Adds a test to check that max_conns_per_ip is enforced

---------

Co-authored-by: teor <teor@riseup.net>
2023-06-19 00:53:00 +00:00
teor 1e12a58b5f
fix(handshake): Add extra timeout logging to peer TCP connections (#6969)
* Add a missing timeout to outbound TCP connections

* Move inbound handshakes into their own function, replacing the manual span

* Delete a useless manual span in zebra_network::config

* Add an extra timeout to the spawned inbound handshake task
2023-06-15 21:11:24 +00:00
Arya 9f0e18282c
fix(logs): Avoid grouping logs for separate tasks (#6923)
* Removes span.enter() in async code

* Restores `accept_span` and uses in_scope() instead of enter()

* Update zebra-network/src/peer_set/initialize.rs
2023-06-14 19:05:11 +00:00
teor 92077f4db5
fix(net): Avoid potential concurrency bugs in outbound handshakes (#6869)
* Stop sending peer errors on the PeerSet channel, to respect send limits

* Move locking out of the cralwer select!, potential deadlock or hang risk

* Move report_failed() out of the CandidateSet, reducing concurrency risks

* Make CandidateSet Send

* Make all CandidateSet operations concurrent, previous hand/deadlock bug

* Reduce the gap between handshakes and peer set updates, and exit the task on shutdown
2023-06-08 23:43:03 +00:00
teor 04e96c2526
feat(net): Cache a list of useful peers on disk (#6739)
* Rewrite some state cache docs to clarify

* Add a zebra_network::Config.cache_dir for peer address caches

* Add new config test files and fix config test failure message

* Create some zebra-chain and zebra-network convenience functions

* Add methods for reading and writing the peer address cache

* Add cached disk peers to the initial peers list

* Add metrics and logging for loading and storing the peer cache

* Replace log of useless redacted peer IP addresses

* Limit the peer cache minimum and maximum size, don't write empty caches

* Add a cacheable_peers() method to the address book

* Add a peer disk cache updater task to the peer set tasks

* Document that the peer cache is shared by multiple instances unless configured otherwise

* Disable peer cache read/write in disconnected tests

* Make initial peer cache updater sleep shorter for tests

* Add unit tests for reading and writing the peer cache

* Update the task list in the start command docs

* Modify the existing persistent acceptance test to check for peer caches

* Update the peer cache directory when writing test configs

* Add a CacheDir type so the default config can be enabled, but tests can disable it

* Update tests to use the CacheDir config type

* Rename some CacheDir internals

* Add config file test cases for each kind of CacheDir config

* Panic if the config contains invalid socket addresses, rather than continuing

* Add a network directory to state cache directory contents tests

* Add new network.cache_dir config to the config parsing tests
2023-06-06 08:28:14 +00:00
teor 0918663e3e
fix(net): Rate-limit MetaAddrChange::Responded from peers (#6738)
* Rate-limit MetaAddrChange::Responded from peers

* Document rate-limits on the address book updater channel
2023-05-23 20:50:29 +00:00
teor b0d9471214
fix(log): Stop logging peer IP addresses, to protect user privacy (#6662)
* Add a PeerSocketAddr type which hides its IP address, but shows the port

* Manually replace SocketAddr with PeerSocketAddr where needed

```sh
fastmod SocketAddr PeerSocketAddr zebra-network
```

* Add missing imports

* Make converting into PeerSocketAddr easier

* Fix some unused imports

* Add a canonical_peer_addr() function

* Fix connection handling for PeerSocketAddr

* Fix serialization for PeerSocketAddr

* Fix tests for PeerSocketAddr

* Remove some unused imports

* Fix address book listener handling

* Remove redundant imports and conversions

* Update outdated IPv4-mapped IPv6 address code

* Make addresses canonical when deserializing

* Stop logging peer addresses in RPC code

* Update zebrad tests with new PeerSocketAddr type

* Update zebra-rpc tests with new PeerSocketAddr type

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-05-14 15:06:07 +00:00
Alfredo Garcia 7c67512cd5
feat(zebra-network): add user agent argument (#6601)
* add user agent as argument, use git to auto build zebra user agent

* try to fix test

* fix typo

* change expect text

* remove newline

* fix some docs

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2023-05-05 00:29:14 +00:00
teor 9c4b3c26fa
fix(net): Reject nodes using ZClassic ports, and warn if configured with those ports (#6567)
* Reject nodes using the ZClassic default ports

* Always check regtest and other coin ports, even if no network is supplied

* Warn if Zebra is configured with ports from other coins

* Allow unspecified addresses and ports for inbound listeners

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-04-28 00:17:37 +00:00
teor 0d50d973d2
fix(net): Limit the number of leftover nonces in the self-connection nonce set (#6534)
* Use a stricter connection rate limit for successful inbound peer connections

* Limit the number of nonces in the self-connection nonce set

* Rate-limit failed inbound connections as well

* Justify the sleep and the yield_now

* Use the configured connection limit rather than a constant

* Tests that the number of nonces is limited (#37)

* Tests that the number of nonces is limited

* removes unused constant

* test that it reaches the nonce limit

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-04-18 08:13:19 +00:00
teor 166526a088
feat(ui): Add a terminal-based progress bar to Zebra (#6235)
* Implement Display and to_string() for NetworkUpgrade

* Add a progress-bar feature to zebrad

* Add the progress bar writer to the tracing component

* Add a block progress bar transmitter

* Correctly shut down the progress bar, and shut it down on an interrupt

* Make it clearer that the progress task never exits

* Add a config for writing logs to a file

* Add a progress-bar feature to zebra-network

* Add a progress bar for the address book size

* Add progress bars for never attempted and failed peers

* Add an optional limit and label to connection counters

* Add open connection progress bars

* Improve CheckpointList API and CheckpointVerifier debugging

* Add checkpoint index and checkpoint queue progress bars

* Security: Limit the number of non-finalized chains tracked by Zebra

* Make some NonFinalizedState methods available with proptest-impl

* Add a non-finalized chain count progress bar

* Track the last fork height for newly forked chains

* Add a should_count_metrics to Chain

* Add a display method for PartialCumulativeWork

* Add a progress bar for each chain fork

* Add a NonFinalizedState::disable_metrics() method and switch to using it

* Move metrics out of Chain because we can't update Arc<Chain>

* Fix: consistently use best chain order when searching chains

* Track Chain progress bars in NonFinalizedState

* Display work as bits, not a multiple of the target difficulty

* Handle negative fork lengths by reporting "No fork"

* Correctly disable unused fork bars

* clippy: rewrite using `match _.cmp(_) { ... }`

* Initial mempool progress bar implementation

* Update Cargo.lock

* Add the actual transaction size as a description to the cost bar

* Only show mempool progress bars after first activation

* Add queued and rejected mempool progress bars

* Clarify cost note is actual size

* Add tracing.log_file config and progress-bar feature to zebrad docs

* Derive Clone for Chain

* Upgrade to howudoin 0.1.2 and remove some bug workarounds

* Directly call the debug formatter to Display a Network

Co-authored-by: Arya <aryasolhi@gmail.com>

* Rename the address count metric to num_addresses

Co-authored-by: Arya <aryasolhi@gmail.com>

* Simplify reverse checkpoint lookup

Co-authored-by: Arya <aryasolhi@gmail.com>

* Simplify progress bar shutdown code

Co-authored-by: Arya <aryasolhi@gmail.com>

* Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE

* Document that the progress task runs forever

* Fix progress log formatting

* If progress-bar is on, log to a file by default

* Create missing directories for log files

* Add file security docs for running Zebra with elevated permissions

* Document automatic log file, spell progress-bar correctly

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-04-13 08:42:17 +00:00
teor c812f880cf
cleanup(clippy): Use inline format strings (#5489)
* Inline format strings using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
cargo fmt --all
```

* Remove unused & and &mut using an automated clippy fix

```sh
cargo clippy --fix --all-features --all-targets -- -A clippy::all -W clippy::uninlined_format_args
```
2022-10-27 13:25:18 +00:00
teor e9c9ea91bc
fix(net): Fix handshake timing and error handling (#4772)
* Actually wait between initial peer connections

* Add a missing span to initial handshake tasks

* Forward handshake panics to the calling task

* Clarify a handshake comment

* Wrap the entire handshake in a timeout, not just some messages

* Actually delay spawning initial connections, so we don't flood the network

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-07-31 21:05:52 +00:00
teor 83b4e6f975
feat(diagnostics): Add tokio-console support to zebrad (#4519)
* Always activate tokio/tracing feature

And always build tests with all tokio features.

* Refactor tracing-subscriber init to simplify it

* Add the tokio-console feature and dependencies

* Add optional tokio-console support, and log the installed tracing layers at info level

Uses a tracing Registry for tokio-console, and a fmt::Subscriber otherwise.

* Add some TODOs based on tracing-subscriber features

* Fix up some spans

* Add a TODO for fixing a log filter bug in tokio-console mode
2022-06-15 16:43:20 +10:00
Marek cc75c3f5f9
fix(doc): Fix various doc warnings, part 3 (#4611)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fa.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

* Fix some rustdoc warnings

* Fix some rustdoc warnings

* Resolve various TODOs

Co-authored-by: teor <teor@riseup.net>

* Fix some unresolved links

* Allow links to private items

* Fix some unresolved links

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-15 03:57:19 +00:00
teor 92fd11f9ad
fix(network): when connecting to peers, ignore invalid ports, and prefer canonical ports (#4564)
* Move peer address validation into its own module

* Add a network parameter to AddressBook and some MetaAddr methods

* Reject invalid initial peers, and connect to them in preferred order

* Reject Flux/ZelCash and misconfigured Zcash peers

* Prefer canonical Zcash ports

* Make peer_preference into a struct method

* Prefer peer addresses with canonical ports for outbound connections

* Also ignore the Zcash regtest port

* Document where field and variant order is required for correctness

* Use the correct peer count

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-14 04:58:37 +00:00
teor ca0520b2e8
change(deps): Upgrade tracing-subscriber and related dependencies (#4517)
* Upgrade tracing and related dependencies

```sh
cargo upgrade --workspace
tracing-error
tracing-subscrber

color-eyre

tracing-flame
tracing-journald

sentry
sentry-tracing

metrics
metrics-exporter-prometheus
reqwest
```

* Update duplicate dependency checks

* Enable the tracing/env-filter feature

* Fix type inference for metrics

Manual changes, plus:
```sh
fastmod "as _" "as f64"
```

* Tidy up some unrelated test code

* Update metrics-exporter-prometheus API

And make unused dependencies optional.

* Adjust test regexes to new tracing format

Also fix some regex bugs, and refactor to simplify.

* Disable color-eyre span traces and track caller in release builds

* Add a feature that enables extra debugging in release builds

* Clean up some redundant features

* Increase a test timeout
2022-06-01 13:53:51 +10:00
teor 6c787dd188
T1. Fix isolated connection bugs, improve tests, upgrade dependencies (#3302)
* Make handshakes generic over AsyncRead + AsyncWrite

* Simplify connect_isolated using ServiceExt::map_err and BoxError

* Move isolated network tests to their own module

* Improve isolated TCP connection tests

* Add an in-memory connection test that uses AsyncReadWrite

* Support connect_isolated on testnet

* Add a wrapper function for isolated TCP connections to an IP address

* Run test tasks for a while, and clean up after them

* Upgrade Zebra dependencies to be compatible with arti, but don't add arti yet

* Fix deny.toml

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2022-01-14 19:34:59 +00:00
teor ac4ed57751
Cancel heartbeats that are waiting for a peer, rather than hanging Zebra (#3325)
* If the crawler is delayed, delay future crawl intervals by the same amount

* Cancel heartbeats that are waiting for network requests or responses
2022-01-12 19:15:07 +00:00
teor 7e63182cdc
Stop ignoring some peers when updating the address book (#3292)
* Make sure MetaAddrChanges are correctly applied to an empty address book

* Add all initial peers to the address book
2022-01-05 18:12:59 -05:00
teor 469fa6b917
1. Fix some address crawler timing issues (#3293)
* Stop holding completed messages until the next inbound message

* Add more info to network message block download debug logs

* Simplify address metrics logs

* Try handling inbound messages as responses, then try as a new request

* Improve address book logging

* Fix a race between the first heartbeat and getaddr requests

* Temporarily reduce the getaddr fanout to 1

* Update metrics when exiting the Connection run loop

* Downgrade some debug logs to trace
2022-01-04 18:43:30 -05:00
teor d0e6de8040
Avoid deadlocks in the address book mutex (#3244)
* Tweak crawler timings so peers are more likely to be available

* Tweak min peer connection interval so we try all peers

* Let other tasks run between fanouts, so we're more likely to choose different peers

* Let other tasks run between retries, so we're more likely to choose different peers

* Let other tasks run after peer crawler DemandDrop

This makes it more likely that peers will become ready.

* Spawn the address book updater on a blocking thread

* Spawn CandidateSet address book operations on blocking threads

* Replace the PeerSet address book with a metrics watch channel

* Fix comment

* Await spawned address book tasks

* Run the address book update tasks concurrently (except for the mutex)

* Explain an internal-only method better

* Fix a typo

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-12-20 00:44:43 +00:00
teor 6cbd7dce43
Fix task handling bugs, so peers are more likely to be available (#3191)
* Tweak crawler timings so peers are more likely to be available

* Tweak min peer connection interval so we try all peers

* Let other tasks run between fanouts, so we're more likely to choose different peers

* Let other tasks run between retries, so we're more likely to choose different peers

* Let other tasks run after peer crawler DemandDrop

This makes it more likely that peers will become ready.
2021-12-20 09:02:31 +10:00
teor ba42d59f12
Stop ignoring panics in inbound handshakes (#3192) 2021-12-10 18:32:42 +00:00
Alfredo Garcia f750535961
Spawn initial handshakes in separated task (#3189)
* spawn connector

* expand comment

Co-authored-by: teor <teor@riseup.net>

* fix error handling

Co-authored-by: teor <teor@riseup.net>
2021-12-10 01:18:43 +00:00
teor 37808eaadb
Security: When there are no new peers, stop crawler using CPU and writing logs (#3177)
* Stop useless crawler attempts when there are no peers and no crawl responses

* Disable GitHub bug report URLs when the disk is full

* Add help text for the `zebrad start` tracing filter option
2021-12-10 00:19:52 +00:00
Janito Vaqueiro Ferreira Filho 0ad89f2f41
Disconnect from outdated peers on network upgrade (#3108)
* Replace usage of `discover::Change` with a tuple

Remove the assumption that a `Remove` variant would never be created
with type changes that allow the compiler to guarantee that assumption.

* Add a `version` field to the `Client` type

Keep track of the peer's reported protocol version.

* Create `LoadTrackedClient` type

A `peer::Client` type wrapper that implements `Load`. This helps with
the creation of a client service that has extra peer information to be
accessed without having to send requests.

* Use `LoadTrackedClient` in `initialize`

Ensure that `PeerSet` receives `LoadTrackedClient`s so that it will be
able to query the peer's protocol version later on.

* Require `LoadTrackedClient` in `PeerSet`

Replace the generic type with a concrete `LoadTrackedClient` so that we
can query its version.

* Create `MinimumPeerVersion` helper type

A type to track the current minimum protocol version for connected
peers based on the current block height.

* Use `MinimumPeerVersion` in handshakes

Keep the code to obtain the current minimum peer protocol version in a
central place.

* Add a `MinimumPeerVersion` instance to `PeerSet`

Prepare it to be able to disconnect from outdated peers based on the
current minimum supported peer protocol version.

* Disconnect from ready services for outdated peers

When the minimum peer protocol version is detected to have changed
(because of a network upgrade), remove all ready services of peers that
became outdated.

* Cancel added unready services of outdated peers

Only add an unready service if it's for a peer that has a supported
protocol version. Otherwise, add it but drop the cancel handle so that
the `UnreadyService` can execute and detect that it was cancelled.

* Avoid adding ready services for outdated peers

If a service becomes ready but it's for a connection to an outdated
peer, drop it.

* Improve comment inside `crawl_and_dial`

Describe an edge case that is also handled but was not explicit.

Co-authored-by: teor <teor@riseup.net>

* Test if calculated minimum peer version is correct

Given an arbitrary best chain tip height, check that the calculated
minimum peer protocol version is the expected value.

* Test if minimum version changes with chain tip

Apply an arbitrary list of chain tip height updates and check that for
each update the minimum peer version is calculated correctly.

* Test minimum peer version changed reports

Simulate a series of best chain tip height updates, and check for
minimum peer version updates at least once between them. Changes should
only be reported once.

* Create a `MockedClientHandle` helper type

Used to create and then track a mock `Client` instance.

* Add `MinimumPeerVersion::with_mock_chain_tip`

An extension method useful for tests, that contains some shared
boilerplate code.

* Bias arbitrary `Version`s to be in valid range

Give a 50% chance for an arbitrary `Version` to be in the range of
previously used values the Zcash network.

* Create a `PeerVersions` helper type

Helps with the creation of mocked client services with arbitrary
protocol versions.

* Create a `PeerSetGuard` helper type

An auxiliary type to a `PeerSet` instance created for testing. It keeps
track of any dummy endpoints of channels created and passed to the
`PeerSet` instance.

* Create a `PeerSetBuilder` helper type

Helps to reduce the code when preparing a `PeerSet` test instance.

* Test if outdated peers are rejected by `PeerSet`

Simulate a set of discovered peers being sent to the `PeerSet`. Ensure
that only up-to-date peers are kept by the `PeerSet` and that outdated
peers are dropped.

* Create `BlockHeightPairAcrossNetworkUpgrades` type

A helper type that allows the creation of arbitrary block height pairs,
where one value is before and the other is at or after the activation
height of an arbitrary network upgrade.

* Test if peers are dropped as they become outdated

Simulate a network upgrade, and check that peers that become outdated
are dropped by the `PeerSet`.

* Remove dbg! macros

Co-authored-by: teor <teor@riseup.net>
2021-12-09 02:54:29 +00:00
teor c4118dcc2c
Check for panics in the address book updater task (#3064)
* Check for panics in the address book updater task

* Fix the return type and tests

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-11-18 12:34:51 +00:00
teor 7d8240fac3
Fix verbose add_initial_peers logs (#3019)
And update some function docs.
2021-11-07 22:21:51 +00:00
Marek d03161c63f
Add unused seed peers to the AddressBook (#2974)
* Add unused seed peers to the AddressBook

* Document a new `await`

We added an extra await on the AddressBook thread mutex.

Co-authored-by: teor <teor@riseup.net>

* Fix a typo

* Refactor names

* Return early from `limit_initial_peers`

* Add `proptest`s regressions

* Return `MetaAddr` instead of `None`

* Test if `zebra_network::init()` deadlocks

* Remove unneeded regressions

* Rename `TimestampCollector` to `AddressBookUpdater` (#2992)

* Rename `TimestampCollector` to `AddressBookUpdater`

* Update comments

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Move `all_peers` instead of copying them

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Make `Duration` a const

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Use a timeout instead of measuring the elapsed time

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Copy `initial_peers` instead of moving them

* Refactor the position of `NewInitial` and `new_initial`

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-04 08:34:00 -03:00
Janito Vaqueiro Ferreira Filho 0960e4fb0b
Update to Tokio 1.13.0 (#2994)
* Update `tower` to version `0.4.9`

Update to latest version to add support for Tokio version 1.

* Replace usage of `ServiceExt::ready_and`

It was deprecated in favor of `ServiceExt::ready`.

* Update Tokio dependency to version `1.13.0`

This will break the build because the code isn't ready for the update,
but future commits will fix the issues.

* Replace import of `tokio::stream::StreamExt`

Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.

* Use `IntervalStream` in `zebra-network`

In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.

* Use `IntervalStream` in `inventory_registry`

In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.

* Use `BroadcastStream` in `inventory_registry`

In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.

* Handle `Semaphore::acquire` error in `tower-batch`

Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.

* Handle `Semaphore::acquire` error in `zebrad` test

On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.

* Update some `zebra-network` dependencies

Use versions compatible with Tokio version 1.

* Upgrade Hyper to version 0.14

Use a version that supports Tokio version 1.

* Update `metrics` dependency to version 0.17

And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.

* Use `f64` as the histogram data type

`u64` isn't supported as the histogram data type in newer versions of
`metrics`.

* Update the initialization of the metrics component

Make it compatible with the new version of `metrics`.

* Simplify build version counter

Remove all constants and use the new `metrics::incement_counter!` macro.

* Change metrics output line to match on

The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.

* Update `sentry` to version 0.23.0

Use a version compatible with Tokio version 1.

* Remove usage of `TracingIntegration`

This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.

* Add sentry layer to tracing initialization

This seems like the replacement for `TracingIntegration`.

* Remove unnecessary conversion

Suggested by a Clippy lint.

* Update Cargo lock file

Apply all of the updates to dependencies.

* Ban duplicate tokio dependencies

Also ban git sources for tokio dependencies.

* Stop allowing sentry-tracing git repository in `deny.toml`

* Allow remaining duplicates after the tokio upgrade

* Use C: drive for CI build output on Windows

GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.

Co-authored-by: teor <teor@riseup.net>
2021-11-02 18:46:57 +00:00
Alfredo Garcia 07610feef3
Reduce outgoing peers demand (#2969)
* reduce demand

* use `saturating_sub`
2021-10-29 16:29:52 +00:00
teor f26a60b801
Limit the number of inbound peer connections (#2961)
* Limit open inbound connections based on the config

* Log inbound connection errors at debug level

* Test inbound connection limits

* Use clone directly in function call argument lists

* Remove an outdated comment

* Update tests to use an unbounded channel rather than mem::forget

And rename some variables.

* Use a lower limit in a slow test and require that it is exceeded
2021-10-28 01:49:31 +00:00
Conrado Gouvea 8d01750459
Rate-limit initial seed peer connections (#2943)
* Rate-limit initial seed peer connections

* Revert "Rate-limit initial seed peer connections"

This reverts commit f779a1eb9e.

* Simplify logic

* Avoid cooperative async task starvation in the peer crawler and listener

If we don't yield in these loops, they can run for a long time before
tokio forces them to yield.

* Add test

* Check for task panics in initial peers test

* Remove duplicate code in rebase

Co-authored-by: teor <teor@riseup.net>
2021-10-27 23:46:43 +00:00
teor 3e03d48799
Limit the number of outbound peer connections (#2944)
* Limit the number of outbound connections in the crawler

* Make zebra-network channel bounds depend on config.peerset_initial_target_size

* Bias Zebra towards outbound connections

And turn connection limits into `Config` methods.

* Downgrade some connection logs to debug

* Remove verbose or outdated fields in tracing logs

* Clarify connection limits

Includes:
- `fastmod OUTBOUND_PEER_BIAS_FRACTION OUTBOUND_PEER_BIAS_DENOMINATOR zebra*`
- clarify connection limit documentation

* Clarify inventory channel capacity

* Add zebra_network::initialize tests with limited numbers of peers

* Avoid cooperative async task starvation in the peer crawler and listener

If we don't yield in these loops, they can run for a long time before
tokio forces them to yield.

* Test the crawler with small connection limits

And use the multi-threaded runtime to avoid long hangs.

* Stop using the multi-threaded executor in tests where it's not needed

* Avoid starvation for every connection

Adds yields after inbound successes and initial peer connections.

* Add a crawler peer connection success test

* Add outbound connection limit tests

* Improve outbound tests
2021-10-27 21:28:51 +00:00
teor c2734f5661
Simplify calling `add_initial_peers` (#2945)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-25 20:16:35 +00:00
teor 67327ac462
Downgrade some less interesting info-level logs to debug (#2938)
There are a lot of these messages when Zebra starts up.
They might be slowing down CI and causing timeouts.

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-10-22 02:11:09 +00:00