* Stop checking the entire AddressBook for each connection attempt
* Stop redundant peer time checks within the address book
* Stop calling `Instant::now` 3 times for each address book update
* Only get the time once each time an address book method is called
* Update outdated comment
* Use an OrderedMap to efficiently store address book peers
* Add address book order tests
Zebra's latest beta continues implementing zero-knowledge proof and note commitment tree validation. In this release, we have finished implementing transaction header, transaction amount, and Zebra-specific NU5 validation. (NU5 mainnet validation is waiting on an `orchard` crate update, and some consensus parameter updates.)
We also fix a number of security issues that could pose a local denial of service risk, or make it easier for an attacker to make a node follow a false chain.
As of this release, Zebra will automatically download and cache the Sprout and Sapling Groth16 circuit parameters. The cache uses around 1 GB of disk space. These cached parameters are shared across all Zebra and `zcashd` instances run by the same user.
See CHANGELOG.md for the full list of changes in this release.
* Tweak a log message
* Only retry failed DNS once, then use the other DNS responses
* Limit broadcasts to half the peers
* Use a longer minimum interval for GetAddr requests
* Reduce the syncer and mempool crawler fanouts
* Stop resetting the mempool twice when it starts up
This spawns two crawlers, which send two fanouts,
so it can use up a lot of peers.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Rewrite PeerSet comments to split long sentences
* Replace peer set integer indexes with address-based indexes
Also improve documentation and logging.
* Security: Stop using peer addresses to choose inventory routing order
* Minor doc and code cleanups
* Stop re-using a drained HashSet
* Replace used `_cancel` with `cancel`
* Reword a comment
* Replace cloned with copied
* Shut down channels and tasks on PeerSet Drop
* Document all the PeerSet fields
* Close the peer set background task handle on shutdown
* Receive background tasks during shutdown
Also, split receiving and polling background tasks into separate methods.
* Revert "Remove commented-out code"
This reverts commit 9e69777925f103ee11e5940bba95b896c828839b.
* Implement deserialization for `addrv2` messages
* Limit addr and addrv2 messages to MAX_ADDRS_IN_MESSAGE
* Clarify address version comments
* Minor cleanups and fixes
* Add preallocation tests for AddrV2
* Add serialization tests for AddrV2
* Use prop_assert in AddrV2 proptests
* Use a generic utility method for deserializing IP addresses in `addrv2`
* Document the purpose of a conversion to MetaAddr
* Fix a comment typo, and clarify that comment
* Clarify the unsupported AddrV2 network ID error and enum variant names
```sh
fastmod AddrV2UnimplementedError UnsupportedAddrV2NetworkIdError zebra-network
fastmod Unimplemented Unsupported zebra-network
```
* Fix and clarify unsupported AddrV2 comments
* Replace `panic!` with `unreachable!`
* Clarify a comment about skipping a length check in a test
* Remove a redundant test
* Basic addr (v1) and addrv2 deserialization tests
* Test deserialized IPv4 and IPv6 values in addr messages
* Remove redundant io::Cursor
* Add comments with expected values of address test vectors
* Add a `Duration32::from_days` constructor
Make it simpler to construct a `Duration32` representing a certain
number of days.
* Add `MetaAddr::was_not_recently_seen` method
A helper method to check if a peer was never seen before or if it was
last seen a long time ago. This will be one of the conditions to
consider a peer as unreachable.
* Add `MetaAddr::is_probably_unreachable` method
A helper method to check if a peer should be considered unreachable. It
is considered unreachable if recent connection attempts have failed and
it was not recently seen.
If a peer is considered unreachable, Zebra shouldn't attempt to connect
to it again.
* Do not keep trying to connect to unreachable peer
A peer is probably unreachable if it was last seen a long time ago and
if it's last connection attempt failed.
* Test `was_not_recently_seen`
Redo the calculation on arbitrary `MetaAddr`s.
* Test `is_probably_unreachable`
Redo the calculation on arbitrary `MetaAddr`s.
* Test if probably unreachable peers are ignored
Given an `AddressBook` with a list of arbitrary `MetaAddr`s, check that
none of the peers listed for a reconnection is probably unreachable.
* Rename unit test to improve clarity
Remove the double negative from the name.
Co-authored-by: teor <teor@riseup.net>
* Rename constant to `MAX_RECENT_PEER_AGE`
Make the purpose of the constant clearer.
Co-authored-by: teor <teor@riseup.net>
* Rename method to `last_seen_is_recent`
Remove the double negative from the name.
* Rename method to `is_probably_reachable`
Avoid having to negate the result of the method in security critical
filter.
* Move check into `is_ready_for_connection_attempt`
Make sure the check is used in any place that requires a peer that's
ready for a connection attempt.
* Improve test documention
Describe the goal of the test better.
Co-authored-by: teor <teor@riseup.net>
* Improve `is_probably_reachable` documentation
List the conditions as bullet points.
Co-authored-by: teor <teor@riseup.net>
* Document what happens when peers have no last seen time
Co-authored-by: teor <teor@riseup.net>
* Implement addr v1 serialization using a separate AddrV1 type
* Remove commented-out code
* Split the address serialization code into modules
* Reorder v1 and in_version fields in serialization order
* Fix a missed search-and-replace
* Explain conversion to MetaAddr
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add unused seed peers to the AddressBook
* Document a new `await`
We added an extra await on the AddressBook thread mutex.
Co-authored-by: teor <teor@riseup.net>
* Fix a typo
* Refactor names
* Return early from `limit_initial_peers`
* Add `proptest`s regressions
* Return `MetaAddr` instead of `None`
* Test if `zebra_network::init()` deadlocks
* Remove unneeded regressions
* Rename `TimestampCollector` to `AddressBookUpdater` (#2992)
* Rename `TimestampCollector` to `AddressBookUpdater`
* Update comments
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Move `all_peers` instead of copying them
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Make `Duration` a const
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use a timeout instead of measuring the elapsed time
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Copy `initial_peers` instead of moving them
* Refactor the position of `NewInitial` and `new_initial`
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Update `tower` to version `0.4.9`
Update to latest version to add support for Tokio version 1.
* Replace usage of `ServiceExt::ready_and`
It was deprecated in favor of `ServiceExt::ready`.
* Update Tokio dependency to version `1.13.0`
This will break the build because the code isn't ready for the update,
but future commits will fix the issues.
* Replace import of `tokio::stream::StreamExt`
Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.
* Use `IntervalStream` in `zebra-network`
In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.
* Use `IntervalStream` in `inventory_registry`
In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.
* Use `BroadcastStream` in `inventory_registry`
In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.
* Handle `Semaphore::acquire` error in `tower-batch`
Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.
* Handle `Semaphore::acquire` error in `zebrad` test
On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.
* Update some `zebra-network` dependencies
Use versions compatible with Tokio version 1.
* Upgrade Hyper to version 0.14
Use a version that supports Tokio version 1.
* Update `metrics` dependency to version 0.17
And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.
* Use `f64` as the histogram data type
`u64` isn't supported as the histogram data type in newer versions of
`metrics`.
* Update the initialization of the metrics component
Make it compatible with the new version of `metrics`.
* Simplify build version counter
Remove all constants and use the new `metrics::incement_counter!` macro.
* Change metrics output line to match on
The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.
* Update `sentry` to version 0.23.0
Use a version compatible with Tokio version 1.
* Remove usage of `TracingIntegration`
This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.
* Add sentry layer to tracing initialization
This seems like the replacement for `TracingIntegration`.
* Remove unnecessary conversion
Suggested by a Clippy lint.
* Update Cargo lock file
Apply all of the updates to dependencies.
* Ban duplicate tokio dependencies
Also ban git sources for tokio dependencies.
* Stop allowing sentry-tracing git repository in `deny.toml`
* Allow remaining duplicates after the tokio upgrade
* Use C: drive for CI build output on Windows
GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.
Co-authored-by: teor <teor@riseup.net>
* Use `prop_assert` instead of `assert`
Otherwise the test input isn't minimized.
* Split long string into a multi-line string
And add some newlines to try to improve readability.
* Fix referenced issue number
They had a typo in their number.
* Make peer services optional
It is unknown for initial peers.
* Fix `preserve_initial_untrusted_values` test
Now that it's optional, the services field can be written to if it was
previously empty.
* Fix formatting of property tests
Run rustfmt on them.
* Restore `TODO` comment
Make it easy to find planned improvements in the code.
Co-authored-by: teor <teor@riseup.net>
* Comment on how ordering is affected
Make it clear that missing services causes the peer to be chosen last.
Co-authored-by: teor <teor@riseup.net>
* Don't expect `services` to be available
Avoid a panic by using the compiler to help enforce the handling of the
case correctly.
* Panic if received gossiped address has no services
All received gossiped addresses have services. The only addresses that
don't have services configured are the initial seed addresses.
Co-authored-by: teor <teor@riseup.net>
* Limit open inbound connections based on the config
* Log inbound connection errors at debug level
* Test inbound connection limits
* Use clone directly in function call argument lists
* Remove an outdated comment
* Update tests to use an unbounded channel rather than mem::forget
And rename some variables.
* Use a lower limit in a slow test and require that it is exceeded
* Rate-limit initial seed peer connections
* Revert "Rate-limit initial seed peer connections"
This reverts commit f779a1eb9e.
* Simplify logic
* Avoid cooperative async task starvation in the peer crawler and listener
If we don't yield in these loops, they can run for a long time before
tokio forces them to yield.
* Add test
* Check for task panics in initial peers test
* Remove duplicate code in rebase
Co-authored-by: teor <teor@riseup.net>
* Limit the number of outbound connections in the crawler
* Make zebra-network channel bounds depend on config.peerset_initial_target_size
* Bias Zebra towards outbound connections
And turn connection limits into `Config` methods.
* Downgrade some connection logs to debug
* Remove verbose or outdated fields in tracing logs
* Clarify connection limits
Includes:
- `fastmod OUTBOUND_PEER_BIAS_FRACTION OUTBOUND_PEER_BIAS_DENOMINATOR zebra*`
- clarify connection limit documentation
* Clarify inventory channel capacity
* Add zebra_network::initialize tests with limited numbers of peers
* Avoid cooperative async task starvation in the peer crawler and listener
If we don't yield in these loops, they can run for a long time before
tokio forces them to yield.
* Test the crawler with small connection limits
And use the multi-threaded runtime to avoid long hangs.
* Stop using the multi-threaded executor in tests where it's not needed
* Avoid starvation for every connection
Adds yields after inbound successes and initial peer connections.
* Add a crawler peer connection success test
* Add outbound connection limit tests
* Improve outbound tests
* Wrap `Sleep` timer in a `Pin<Box<_>>`
The `Sleep` type doesn't implement `Unpin` in newer versions of Tokio.
* Wrap `Sleep` type in a `Pin<Box<_>>`
In newer Tokio versions the `Sleep` type doesn't implement `Unpin`, so
it needs to be manually pinned.
There are a lot of these messages when Zebra starts up.
They might be slowing down CI and causing timeouts.
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Replace some unit tuples with named unit structs
This helps distinguish generic channels and make them type-safe.
Also tidy imports and documentation in `peer_set::set`.
* Link to the tower balance crate from docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* limit the number of initial peers
* Move more code out of zebra_network::initialize
* Always limit the number of initial peers in the Config
This way, we can never get the unused peers out.
* Revert "Always limit the number of initial peers in the Config"
This reverts commit 81ede597c8.
Actually, this doesn't work, because we want those extra peers.
* Minor tweaks
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Count the number of active inbound and outbound peer connections
And reduce the count when each connection fails.
* Fix a comment typo
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Newer versions of Tokio panic if `tokio::time::pause()` is called from a
multi-thread executor, and `#[tokio::test]` defaults to a single thread
runtime, so it makes sense to always use a single thread runtime in all
tests.
In newer Tokio versions the `Sleep` type doesn't implement `Unpin`, so
it's a little more complicated to use it. In this case it was easier to
refactor the code to not store the `Sleep` type instead of wrapping it
in a `Pin` type.
* Increment the crates that have new commits since the last version
* Increment the crates that depend on crates that have changed
* Increment the version of `zebra-script`
* Use the `zebrad` version in the `zebra-network` user agent string
* Use the `v1.0.0-alpha.19` git tag in `README.md`
* Copy the draft changelog into `CHANGELOG.md`
* Delete bumps
* Update CHANGELOG.md
Co-authored-by: teor <teor@riseup.net>
* Add newly merged PRs
Co-authored-by: teor <teor@riseup.net>
* Improve logging for initial peer connections
* Decrease the initial peer crawl timeout to make tests more reliable
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Update some comments
* Add a mempool debug_enable_at_height config
* Rename a field in the mempool crawler
* Propagate syncer channel errors through the crawler
We don't want to ignore these errors, because they might indicate a shutdown.
(Or a bug that we should fix.)
* Use debug_enable_at_height in the mempool crawler
* Log when the mempool is activated or deactivated
* Deny unknown fields and apply defaults for all configs
* Move Duration last, as required for TOML tables
* Add a basic mempool acceptance test
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Update versions for zebra v1.0.0-alpha.18 release
* WIP: Initial PR list
* Remove uninteresting version bumps from CHANGELOG
* Categorise and group PRs in CHANGELOG, removing uninteresting PRs
* Further refine and categorise changelog entries
* Fix tag url
* Final changes to CHANGELOG
* Add a changelog description
* Spacing
* Clarify and fix changelog PR descriptions
* Add PRs that are about to be merged
* More slight clarifications
* Spacing
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Check return value of zcash_script_new_precomputed_tx
* Set the NU5 testnet activation height to 1_590_000
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update Nu5 constants to new values
* Update ZIP-244 test vectors for new branch ID
* Squashed commit of the following:
commit bdb120a249
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 11:54:01 2021 -0400
Use pallas::Base::from_str_vartime() in sinsemilla tests
commit e99fa49258
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 11:45:24 2021 -0400
Compiles
commit a520018114
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 10:15:17 2021 -0400
Incomplete upgrade of deps
* Squashed commit of the following:
commit 8d1b76ec5626517817c3a4d9f3950acc90a359df
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 04:02:26 2021 +0000
Update `zcash_script` to support V5 transactions
Use a newer version of `zcash_script` that has been updated to support
V5 transactions.
commit 371233628ae61e0c25d6ba8f31d9dba42823becb
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 03:06:20 2021 +0000
Update Zcash dependencies
Update some Zcash crates:
- `halo2`
- `incrementalmerkletree' (patch version)
- `orchard` (patch version)
- `zcash_history` (patch version)
- `zcash_note_encryption` (patch version)
- `zcash_primitives` (patch version)
And also update the `group` dependency so that the code remains
compatible.
commit de5cf1ec40c3fc08670fc971cdf3e65e13d9f4c7
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 03:04:13 2021 +0000
Update error message assertion
Use the updated message for the expected error variant.
* Update `zcash_script` to support V5 transactions
Use a newer version of `zcash_script` that has been updated to support
V5 transactions.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Add metrics gauges for the most recent peer network protocol version
This gague lets us join the initial seeds to the network protocol versions,
even if the peer upgrades and reconnects with a different version.
* Ensure dashboard peer network versions are unique
Otherwise, prometheus returns an error,
and the dashboard shows no data.
* Make seeder labels more readable
- put labels to the right of the graph
- remove default ports
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
teor
Pili Guerra
Dimitris Apostolou
Janito Vaqueiro Ferreira Filho
Marek
Conrado Gouvea
Alfredo Garcia