Commit Graph

38 Commits

Author SHA1 Message Date
Alfredo Garcia 887d4a2457
fix(beta-toolchain): Warnings (#8404)
* fix beta imports and other warnings

* clippy lints beta

* move attributes to after docs

Co-authored-by: Marek <mail@marek.onl>

* move some code to test module

* implement display instead of direct tostring for LongPollId

Co-authored-by: Arya <aryasolhi@gmail.com>

* fix typo

---------

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
2024-04-18 01:45:22 +00:00
idky137 4579722833
change(chain): Remove `Copy` trait impl from `Network` (#8354)
* removed derive copy from network, address and ledgerstate

* changed is_max_block_time_enforced to accept ref

* changed NetworkUpgrade::Current to accept ref

* changed NetworkUpgrade::Next to accept ref

* changed NetworkUpgrade::IsActivationHeight to accept ref

* changed NetworkUpgrade::TargetSpacingForHeight to accept ref

* changed NetworkUpgrade::TargetSpacings to accept ref

* changed NetworkUpgrade::MinimumDifficultySpacing_forHeight to accept ref

* changed NetworkUpgrade::IsTestnetMinDifficultyBlock to accept ref

* changed NetworkUpgrade::AveragingWindowTimespanForHeight to accept ref

* changed NetworkUpgrade::ActivationHeight to accept ref

* changed sapling_activation_height to accept ref

* fixed lifetime for target_spacings

* fixed sapling_activation_height

* changed transaction_to_fake_v5 and fake_v5_transactions_for_network to accept ref to network

* changed Input::vec_strategy to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_history.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/history_tree.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/viewing_key* to accept ref to network

* changed functions in zebra-chain/src/transparent/address.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_primitives.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/zcash_note_encryption.rs to accept ref to network

* changed functions in zebra-chain/src/primitives/history_tree* to accept ref to network

* changed functions in zebra-chain/src/block* to accept ref to network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain::parameters::network

* fixed errors in zebra-chain

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* changed NonEmptyHistoryTree and InnerHistoryTree to hold value instead of ref

* fixed errors in zebra-chain/src/block/arbitrary.rs

* finished fixing errors in zebra-chain - all crate tests pass

* changed functions in zebra-state::service::finalized_state to accept &Network

* changed functions in zebra-state::service::non_finalized_state to accept &Network

* zebra-state tests run but fail with overflow error

* zebra-state tests all pass

* converted zebra-network -- all crate tests pass

* applied all requested changes from review

* converted zebra-consensus -- all crate tests pass

* converted zebra-scan -- all crate tests pass

* converted zebra-rpc -- all crate tests pass

* converted zebra-grpc -- all crate tests pass

* converted zebrad -- all crate tests pass

* applied all requested changes from review

* fixed all clippy errors

* fixed build error in zebrad/src/components/mempool/crawler.rs
2024-03-19 20:45:27 +00:00
teor 4aaeb4c4cc
cleanup(net): Deleted unused NeverAttemptedAlternate peer state (#7982)
* Update Connection debug impl missed in previous PRs

* Add an initial_cached_addrs argument to Connection::new()

* Stop sending version and remote IP peers directly to the address book

* Remove the NeverAttemptedAlternate peer state

* Remove the NewAlternate MetaAddrChange

* Remove the never_attempted_alternate AddressBookMetrics

* Update CandidateSet docs and add a Debug impl
2023-11-28 00:30:13 +00:00
teor 43e54d1cb2
fix(net): Fix a potential hang caused by accessing the address book directly (#7902)
* Fix a potential hang accessing the address book directly

* Remove unused connection shutdown MetaAddr arguments

* Add an UpdateConnected MetaAddrChange, that sends initial connection info

* Fix some tests

* Fix a panic with a zero channel size
2023-11-05 22:28:58 +00:00
Arya 77ad91ced4
fix(network): Avoid initiating outbound handshakes with IPs for which Zebra already has an active peer. (#7029)
* Adds most_recent_by_ip field to address book

* adds test

* Apply suggestions from code review

* fixes lint

* Updates most_recent_by_ip in .take()

Updates should_update_most_recent_by_ip() and has_active_peer_with_ip to check last_attempt and last_failure times

Renames has_active_peer_with_ip

* Documents that Zebra will not initiate more than 1 outbound connections per IP

* Fixes is_ready_for_connection_attempt_with_ip

Adds test coverage for AttemptPending and Failed

Fixes new_outbound_peer_connections_are_rate_limited proptest

* Applies suggestions from code review.

* Applies suggestions from code review

* Always return true from `is_ready_for_connection_attempt_with_ip` if max_connections_per_ip != 0

* Update max_connections_per_ip config docs

* Warn about invalid config fields and use default values

* Ignores last_attempt and last_failure in is_ready_for_connection_attempt_with_ip

updates test

* Only update most_recent_by_ip if update.last_conn_state is responded.

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* fixes lint

* Update zebra-network/src/address_book.rs

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Fix Rust syntax

* Fix whitespace

---------

Co-authored-by: teor <teor@riseup.net>
2023-07-06 05:54:10 +00:00
teor e6c3b87872
Stop panicking on shutdown in the syncer and network init (#7104) 2023-07-02 20:08:11 +00:00
teor 04e96c2526
feat(net): Cache a list of useful peers on disk (#6739)
* Rewrite some state cache docs to clarify

* Add a zebra_network::Config.cache_dir for peer address caches

* Add new config test files and fix config test failure message

* Create some zebra-chain and zebra-network convenience functions

* Add methods for reading and writing the peer address cache

* Add cached disk peers to the initial peers list

* Add metrics and logging for loading and storing the peer cache

* Replace log of useless redacted peer IP addresses

* Limit the peer cache minimum and maximum size, don't write empty caches

* Add a cacheable_peers() method to the address book

* Add a peer disk cache updater task to the peer set tasks

* Document that the peer cache is shared by multiple instances unless configured otherwise

* Disable peer cache read/write in disconnected tests

* Make initial peer cache updater sleep shorter for tests

* Add unit tests for reading and writing the peer cache

* Update the task list in the start command docs

* Modify the existing persistent acceptance test to check for peer caches

* Update the peer cache directory when writing test configs

* Add a CacheDir type so the default config can be enabled, but tests can disable it

* Update tests to use the CacheDir config type

* Rename some CacheDir internals

* Add config file test cases for each kind of CacheDir config

* Panic if the config contains invalid socket addresses, rather than continuing

* Add a network directory to state cache directory contents tests

* Add new network.cache_dir config to the config parsing tests
2023-06-06 08:28:14 +00:00
teor 8af4e572c9
fix(network): Ignore out of order Address Book changes, unless they are concurrent (#6717)
* Ignore out of order Address Book changes, and restructure the function

* Handle concurrent changes using the connection state machine order

* Handle out of order changes correctly

* Pass times through the call stack so they are consistent in tests

* Add time arguments to tests

* Fix tests that were broken by the address order checks

* fastmod wall_ local_ zebra*

* cargo fmt --all

* Fix a bug in the concurrent change check

* Test all the new apply and skip checks for address changes

* Document more edge cases and increase the concurrency time slightly

* Simplify enum ordering matches

* Fix comment typos

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-05-24 23:53:53 +00:00
teor b0d9471214
fix(log): Stop logging peer IP addresses, to protect user privacy (#6662)
* Add a PeerSocketAddr type which hides its IP address, but shows the port

* Manually replace SocketAddr with PeerSocketAddr where needed

```sh
fastmod SocketAddr PeerSocketAddr zebra-network
```

* Add missing imports

* Make converting into PeerSocketAddr easier

* Fix some unused imports

* Add a canonical_peer_addr() function

* Fix connection handling for PeerSocketAddr

* Fix serialization for PeerSocketAddr

* Fix tests for PeerSocketAddr

* Remove some unused imports

* Fix address book listener handling

* Remove redundant imports and conversions

* Update outdated IPv4-mapped IPv6 address code

* Make addresses canonical when deserializing

* Stop logging peer addresses in RPC code

* Update zebrad tests with new PeerSocketAddr type

* Update zebra-rpc tests with new PeerSocketAddr type

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-05-14 15:06:07 +00:00
Conrado Gouvea 6fd750e168
build(deps): bump insta from 1.15.0 to 1.17.1 (#4884)
* build(deps): bump insta from 1.15.0 to 1.17.1

Bumps [insta](https://github.com/mitsuhiko/insta) from 1.15.0 to 1.17.1.
- [Release notes](https://github.com/mitsuhiko/insta/releases)
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/insta/compare/1.15.0...1.17.1)

---
updated-dependencies:
- dependency-name: insta
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* make zebra_test::init() return the insta drop guard

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-08-04 15:44:44 +00:00
teor 92fd11f9ad
fix(network): when connecting to peers, ignore invalid ports, and prefer canonical ports (#4564)
* Move peer address validation into its own module

* Add a network parameter to AddressBook and some MetaAddr methods

* Reject invalid initial peers, and connect to them in preferred order

* Reject Flux/ZelCash and misconfigured Zcash peers

* Prefer canonical Zcash ports

* Make peer_preference into a struct method

* Prefer peer addresses with canonical ports for outbound connections

* Also ignore the Zcash regtest port

* Document where field and variant order is required for correctness

* Use the correct peer count

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-14 04:58:37 +00:00
Marek 2e50ccc8f3
fix(doc): Fix various doc warnings, part 2 (#4561)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fa.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

* Fix some rustdoc warnings

* Fix some rustdoc warnings

* Resolve various TODOs

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-06-14 01:22:16 +00:00
teor a4dd3b7396
4. Avoid repeated requests to peers after partial responses or errors (#3505)
* fix(network): split synthetic NotFoundRegistry from message NotFoundResponse

* docs(network): Improve `notfound` message documentation

* refactor(network): Rename MustUseOneshotSender to MustUseClientResponseSender

```
fastmod MustUseOneshotSender MustUseClientResponseSender zebra*
```

* docs(network): fix a comment typo

* refactor(network): remove generics from MustUseClientResponseSender

* refactor(network): add an inventory collector to Client, but don't use it yet

* feat(network): register missing peer responses as missing inventory

We register this missing inventory based on peer responses,
or connection errors or timeouts.

Inbound message inventory tracking requires peers to send `notfound` messages.
But `zcashd` skips `notfound` for blocks, so we can't rely on peer messages.
This missing inventory tracking works regardless of peer `notfound` messages.

* refactor(network): rename ResponseStatus to InventoryResponse

```sh
fastmod ResponseStatus InventoryResponse zebra*
```

* refactor(network): rename InventoryStatus::inner() to to_inner()

* fix(network): remove a redundant runtime.enter() in a test

* doc(network): the exact time used to filter outbound peers doesn't matter

* fix(network): handle block requests slightly more efficiently

* doc(network): fix a typo

* fmt(network): `cargo fmt` after rename ResponseStatus to InventoryResponse

* doc(test): clarify some test comments

* test(network): test synthetic notfound from connection errors and peer inventory routing

* test(network): improve inbound test diagnostics

* feat(network): add a proptest-impl feature to zebra-network

* feat(network): add a test-only connect_isolated_with_inbound function

* test(network): allow a response on the isolated peer test connection

* test(network): fix failures in test synthetic notfound

* test(network): Simplify SharedPeerError test assertions

* test(network): test synthetic notfound from partially successful requests

* test(network): MissingInventoryCollector ignores local NotFoundRegistry errors

* fix(network): decrease the inventory rotation interval

This stops us waiting 3-4 sync resets (4 minutes) before we retry a missing block.

Now we wait 1-2 sync resets (2 minutes), which is still a reasonable rate limit.
This should speed up syncing near the tip, and on testnet.

* fmt(network): cargo fmt --all

* cleanup(network): remove unnecessary allow(dead_code)

* cleanup(network): stop importing the whole sync module into tests

* doc(network): clarify syncer inventory retry constraint

* doc(network): add a TODO for a fix to ensure API behaviour remains consistent

* doc(network): fix a function doc typo

* doc(network): clarify how we handle peers that don't send `notfound`

* docs(network): clarify a test comment

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-15 01:44:33 +00:00
teor 7e63182cdc
Stop ignoring some peers when updating the address book (#3292)
* Make sure MetaAddrChanges are correctly applied to an empty address book

* Add all initial peers to the address book
2022-01-05 18:12:59 -05:00
teor 0f92b6729a
Fix an unrelated type resolution error (#3304) 2021-12-29 23:44:28 +00:00
teor 4ce6fbccc4
Fix new clippy lints in clippy nightly (#3176) 2021-12-09 14:19:14 +00:00
Janito Vaqueiro Ferreira Filho 1f756fcc81
Add `zebra_test::init_async` helper function (#3169)
* Use a single-thread shared Tokio runtime

This allows it to pause the time and more closely resembles the
environment that's set by default for asynchronous tests.

* Add a `zebra_test::init_async` helper function

Calls `zebra_test::init` but also constructs a single-thread Tokio
runtime and returns it. This makes it simpler to initialize asynchronous
tests that can't use the `#[tokio::test]` attribute.

* Replace usages of `Runtime::new` in tests

Use the new `zebra_test::init_async()` helper function instead.

* Replace `runtime::Builder::new_current_thread()`

Use the new `zebra_test::init_async()` helper function instead.

* Replace `runtime::Builder::new_multi_thread()`

Use the new `zebra_test::init_async()` helper function instead. The test
with the change doesn't necessarily have to use a multi-thread runtime.
2021-12-09 00:18:17 +00:00
teor 332afc17d5
Security: Limit address book size to limit memory usage (#3162)
* Refactor the address response limit

* Limit the number of peers in the address book

* Allow changing the address book limit in tests

* Add tests for the address book length limit

* rustfmt
2021-12-06 16:09:10 -03:00
teor 4d608d3224
Stop doing thousands of time checks each time we connect to a peer (#3106)
* Stop checking the entire AddressBook for each connection attempt

* Stop redundant peer time checks within the address book

* Stop calling `Instant::now` 3 times for each address book update

* Only get the time once each time an address book method is called

* Update outdated comment

* Use an OrderedMap to efficiently store address book peers

* Add address book order tests
2021-12-03 15:09:43 -03:00
Janito Vaqueiro Ferreira Filho 11b5a33651
Security: Avoid reconnecting to peers that are likely unreachable (#3030)
* Add a `Duration32::from_days` constructor

Make it simpler to construct a `Duration32` representing a certain
number of days.

* Add `MetaAddr::was_not_recently_seen` method

A helper method to check if a peer was never seen before or if it was
last seen a long time ago. This will be one of the conditions to
consider a peer as unreachable.

* Add `MetaAddr::is_probably_unreachable` method

A helper method to check if a peer should be considered unreachable. It
is considered unreachable if recent connection attempts have failed and
it was not recently seen.

If a peer is considered unreachable, Zebra shouldn't attempt to connect
to it again.

* Do not keep trying to connect to unreachable peer

A peer is probably unreachable if it was last seen a long time ago and
if it's last connection attempt failed.

* Test `was_not_recently_seen`

Redo the calculation on arbitrary `MetaAddr`s.

* Test `is_probably_unreachable`

Redo the calculation on arbitrary `MetaAddr`s.

* Test if probably unreachable peers are ignored

Given an `AddressBook` with a list of arbitrary `MetaAddr`s, check that
none of the peers listed for a reconnection is probably unreachable.

* Rename unit test to improve clarity

Remove the double negative from the name.

Co-authored-by: teor <teor@riseup.net>

* Rename constant to `MAX_RECENT_PEER_AGE`

Make the purpose of the constant clearer.

Co-authored-by: teor <teor@riseup.net>

* Rename method to `last_seen_is_recent`

Remove the double negative from the name.

* Rename method to `is_probably_reachable`

Avoid having to negate the result of the method in security critical
filter.

* Move check into `is_ready_for_connection_attempt`

Make sure the check is used in any place that requires a peer that's
ready for a connection attempt.

* Improve test documention

Describe the goal of the test better.

Co-authored-by: teor <teor@riseup.net>

* Improve `is_probably_reachable` documentation

List the conditions as bullet points.

Co-authored-by: teor <teor@riseup.net>

* Document what happens when peers have no last seen time

Co-authored-by: teor <teor@riseup.net>
2021-11-10 23:51:22 +00:00
teor 85b016756d
Refactor addr v1 serialization using a separate AddrV1 type (#3021)
* Implement addr v1 serialization using a separate AddrV1 type

* Remove commented-out code

* Split the address serialization code into modules

* Reorder v1 and in_version fields in serialization order

* Fix a missed search-and-replace

* Explain conversion to MetaAddr

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-11-10 06:47:50 +10:00
teor 88cdf0f832
Move MetaAddr serialization into zebra_network::protocol::external (#3020)
Preparation for `addrv2` serialization.
2021-11-07 21:37:38 +00:00
Janito Vaqueiro Ferreira Filho a9f1c189d9
Make `services` field in `MetaAddr` optional (#2976)
* Use `prop_assert` instead of `assert`

Otherwise the test input isn't minimized.

* Split long string into a multi-line string

And add some newlines to try to improve readability.

* Fix referenced issue number

They had a typo in their number.

* Make peer services optional

It is unknown for initial peers.

* Fix `preserve_initial_untrusted_values` test

Now that it's optional, the services field can be written to if it was
previously empty.

* Fix formatting of property tests

Run rustfmt on them.

* Restore `TODO` comment

Make it easy to find planned improvements in the code.

Co-authored-by: teor <teor@riseup.net>

* Comment on how ordering is affected

Make it clear that missing services causes the peer to be chosen last.

Co-authored-by: teor <teor@riseup.net>

* Don't expect `services` to be available

Avoid a panic by using the compiler to help enforce the handling of the
case correctly.

* Panic if received gossiped address has no services

All received gossiped addresses have services. The only addresses that
don't have services configured are the initial seed addresses.

Co-authored-by: teor <teor@riseup.net>
2021-11-02 02:45:35 +00:00
Janito Vaqueiro Ferreira Filho 39ed7d70d3
Use single thread Tokio runtime for tests (#2916)
Newer versions of Tokio panic if `tokio::time::pause()` is called from a
multi-thread executor, and `#[tokio::test]` defaults to a single thread
runtime, so it makes sense to always use a single thread runtime in all
tests.
2021-10-21 16:22:12 +00:00
Janito Vaqueiro Ferreira Filho b68202c68a
Security: Zebra should stop gossiping unreachable addresses to other nodes, Action: re-deploy all nodes (#2392)
* Rename some methods and constants for clarity

Using the following commands:

```
fastmod '\bis_ready_for_attempt\b' is_ready_for_connection_attempt
  # One instance required a tweak, because of the ASCII diagram.
fastmod '\bwas_recently_live\b' has_connection_recently_responded
fastmod '\bwas_recently_attempted\b' was_connection_recently_attempted
fastmod '\bwas_recently_failed\b' has_connection_recently_failed
fastmod '\bLIVE_PEER_DURATION\b' MIN_PEER_RECONNECTION_DELAY
```

* Use `Instant::elapsed` for conciseness

Instead of `Instant::now().saturating_duration_since`. They're both
equivalent, and `elapsed` only panics if the `Instant` is somehow
synthetically generated.

* Allow `Duration32` to be created in other crates

Export the `Duration32` from the `zebra_chain::serialization` module.

* Add some new `Duration32` constructors

Create some helper `const` constructors to make it easy to create
constant durations. Add methods to create a `Duration32` from seconds,
minutes and hours.

* Avoid gossiping unreachable peers

When sanitizing the list of peers to gossip, remove those that we
haven't seen in more than three hours.

* Test if unreachable addresses aren't gossiped

Create a property test with random addreses inserted into an
`AddressBook`, and verify that the sanitized list of addresses does not
contain any addresses considered unreachable.

* Test if new alternate address isn't gossipable

Create a new alternate peer, because that type of `MetaAddr` does not
have `last_response` or `untrusted_last_seen` times. Verify that the
peer is not considered gossipable.

* Test if local listener is gossipable

The `MetaAddr` representing the local peer's listening address should
always be considered gossipable.

* Test if gossiped peer recently seen is gossipable

Create a `MetaAddr` representing a gossiped peer that was reported to be
seen recently. Check that the peer is considered gossipable.

* Test peer reportedly last seen in the future

Create a `MetaAddr` representing a peer gossiped and reported to have
been last seen in a time that's in the future. Check that the peer is
considered gossipable, to check that the fallback calculation is working
as intended.

* Test gossiped peer reportedly seen long ago

Create a `MetaAddr` representing a gossiped peer that was reported to
last have been seen a long time ago. Check that the peer is not
considered gossipable.

* Test if just responded peer is gossipable

Create a `MetaAddr` representing a peer that has just responded and
check that it is considered gossipable.

* Test if recently responded peer is gossipable

Create a `MetaAddr` representing a peer that last responded within the
duration a peer is considered reachable. Verify that the peer is
considered gossipable.

* Test peer that responded long ago isn't gossipable

Create a `MetaAddr` representing a peer that last responded outside the
duration a peer is considered reachable. Verify that the peer is not
considered gossipable.
2021-06-29 05:12:27 +00:00
teor bcd5f2c50d
Gossip dynamic local listener ports to peers (#2277)
* Gossip dynamically allocated listener ports to peers

Previously, Zebra would either gossip port `0`, which is invalid, or skip
gossiping its own dynamically allocated listener port.

* Improve "no configured peers" warning

And downgrade from error to warning, because inbound-only nodes are a
valid use case.

* Move random_known_port to zebra-test

* Add tests for dynamic local listener ports and the AddressBook

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-06-23 07:59:06 +10:00
teor 1a57023eac
Security: Use canonical SocketAddrs to avoid duplicate peer connections, Feature: Send local listener to peers (#2276)
* Always send our local listener with the latest time

Previously, whenever there was an inbound request for peers, we would
clone the address book and update it with the local listener.

This had two impacts:
- the listener could conflict with an existing entry,
  rather than unconditionally replacing it, and
- the listener was briefly included in the address book metrics.

As a side-effect, this change also makes sanitization slightly faster,
because it avoids some useless peer filtering and sorting.

* Skip listeners that are not valid for outbound connections

* Filter sanitized addresses Zebra based on address state

This fix correctly prevents Zebra gossiping client addresses to peers,
but still keeps the client in the address book to avoid reconnections.

* Add a full set of DateTime32 and Duration32 calculation methods

* Refactor sanitize to use the new DateTime32/Duration32 methods

* Security: Use canonical SocketAddrs to avoid duplicate connections

If we allow multiple variants for each peer address, we can make multiple
connections to that peer.

Also make sure sanitized MetaAddrs are valid for outbound connections.

* Test that address books contain the local listener address

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-06-22 02:16:59 +00:00
teor 4d22a0bae9
Security: Limit reconnection rate to individual peers (#2275)
* Security: Limit reconnection rate to individual peers

Reconnection Rate

Limit the reconnection rate to each individual peer by applying the
liveness cutoff to the attempt, responded, and failure time fields.
If any field is recent, the peer is skipped.

The new liveness cutoff skips any peers that have recently been attempted
or failed. (Previously, the liveness check was only applied if the peer
was in the `Responded` state, which could lead to repeated retries of
`Failed` peers, particularly in small address books.)

Reconnection Order

Zebra prefers more useful peer states, then the earliest attempted,
failed, and responded times, then the most recent gossiped last seen
times.

Before this change, Zebra took the most recent time in all the peer time
fields, and used that time for liveness and ordering. This led to
confusion between trusted and untrusted data, and success and failure
times.

Unlike the previous order, the new order:
- tries all peers in each state, before re-trying any peer in that state,
  and
- only checks the the gossiped untrusted last seen time
  if all other times are equal.

* Preserve the later time if changes arrive out of order

* Update CandidateSet::next documentation

* Update CandidateSet state diagram

* Fix variant names in comments

* Explain why timestamps can be left out of MetaAddrChanges

* Add a simple test for the individual peer retry limit

* Only generate valid Arbitrary PeerServices values

* Add an individual peer retry limit AddressBook and CandidateSet test

* Stop deleting recently live addresses from the address book

If we delete recently live addresses from the address book, we can get a
new entry for them, and reconnect too rapidly.

* Rename functions to match similar tokio API

* Fix docs for service sorting

* Clarify a comment

* Cleanup a variable and comments

* Remove blank lines in the CandidateSet state diagram

* Add a multi-peer proptest that checks outbound attempt fairness

* Fix a comment typo

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Simplify time maths in MetaAddr

* Create a Duration32 type to simplify calculations and comparisons

* Rename variables for clarity

* Split a string constant into multiple lines

* Make constants match rustdoc order

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-06-18 09:30:44 -03:00
teor 3f7410d073
Security: stop gossiping failure and attempt times as last_seen times (#2273)
* Security: stop gossiping failure and attempt times as last_seen times

Previously, Zebra had a single time field for peer addresses, which was
updated every time a peer was attempted, sent a message, or failed.

This is a security issue, because the `last_seen` time should be
"the last time [a peer] connected to that node", so that
"nodes can use the time field to avoid relaying old 'addr' messages".
So Zebra was sending incorrect peer information to other nodes.

As part of this change, we split the `last_seen` time into the
following fields:
- untrusted_last_seen: gossiped from other peers
- last_response: time we got a response from a directly connected peer
- last_attempt: time we attempted to connect to a peer
- last_failure: time a connection with a peer failed

* Implement Arbitrary and strategies for MetaAddrChange

Also replace the MetaAddr Arbitrary impl with a derive.

* Write proptests for MetaAddr and MetaAddrChange

MetaAddr:
- the only times that get included in serialized MetaAddrs are
  the untrusted last seen and responded times

MetaAddrChange:
- the untrusted last seen time is never updated
- the services are only updated if there has been a handshake
2021-06-15 13:31:16 +10:00
teor 86f23f7960
Security: only apply the outbound connection rate-limit to actual connections (#2278)
* Only advance the outbound connection timer when it returns an address

Previously, we were advancing the timer even when we returned `None`.
This created large wait times when there were no eligible peers.

* Refactor to avoid overlapping sleep timers

* Add a maximum next peer delay test

Also refactor peer numbers into constants.

* Make the number of proptests overridable by the standard env var

Also cleanup the test constants.

* Test that skipping peer connections also skips their rate limits

* Allow an extra second after each sleep on loaded machines

macOS VMs seem to need this extra time to pass their tests.

* Restart test time bounds from the current time

This change avoids test failures due to cumulative errors.

Also use a single call to `Instant::now` for each test round.
And print the times when the tests fail.

* Stop generating invalid outbound peers in proptests

The candidate set proptests will fail if enough generated peers are
invalid for outbound connections.
2021-06-15 08:29:17 +10:00
Janito Vaqueiro Ferreira Filho aaef94c2bf
Prevent burst of reconnection attempts (#2251)
* Rate-limit new outbound peer connections

Set the rate-limiting sleep timer to use a delay added to the maximum
between the next peer connection instant and now. This ensures that the
timer always sleeps at least the time used for the delay.

This change fixes rate-limiting new outbound peer connections, since
before there could be a burst of attempts until the deadline progressed
to the current instant.

Fixes #2216

* Create `MetaAddr::alternate_node_strategy` helper

Creates arbitrary `MetaAddr`s as if they were network nodes that sent
their listening address.

* Test outbound peer connection rate limiting

Tests if connections are rate limited to 10 per second, and also tests
that sleeping before continuing with the attempts still respets the rate
limit and does not result in a burst of reconnection attempts.
2021-06-07 14:13:46 +10:00
teor ebe1c9f88e
Add a DateTime32 type for 32-bit serialized times (#2210)
* Add a DateTime32 type for 32-bit serialized times
* Use DateTime32 for MetaAddr.last_seen
* Create and use a `DateTime32::now` method
2021-05-31 12:52:34 +10:00
teor 5cdcc5255f Proptest `MetaAddr` sanitization and serialization together 2021-05-26 18:13:35 -04:00
teor 9f8b4f836e Test round-trip serialization for gossiped `MetaAddr`s 2021-05-26 18:13:35 -04:00
teor 078385ae00 Canonicalise arbitrary IP addresses in proptests
This makes round-trip serialization tests work.
2021-05-26 18:13:35 -04:00
teor b600e82d6e
Security: Avoid silently corrupting invalid times during serialization (#2149)
* Security: panic if an internally generated time is out of range

If Zebra has a bug where it generates blocks, transactions, or meta
addresses with bad times, panic. This avoids sending bad data onto the
network.

(Previously, Zebra would truncate some of these times, silently
corrupting the underlying data.)

Make it clear that deserialization of these objects is infalliable.
2021-05-17 16:53:10 -04:00
teor 1626ec383a
Add InventoryHash and MetaAddr proptests (#1985)
* Make proptest dependencies consistent between chain and network

* Implement Arbitrary for InventoryHash and use it in tests

* Impl Arbitrary for MetaAddr and use it in tests

Also test some extreme times in MetaAddr sanitization.
2021-04-07 14:13:52 -03:00
teor 64662a758d
Move the preallocate tests into their own files (#1977)
* Move the preallocate tests into their own files

And move the MetaAddr proptest into its own file.

Also do some minor formatting and cleanups.

Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
2021-04-07 12:32:27 +10:00