* Add CompactSize64 and CompactSizeMessage types
But don't remove read_compactsize and write_compactsize yet.
* Fix CompactSize capitalisation
```sh
fastmod compactSize CompactSize zebra* book
fastmod compactsize CompactSize zebra* book
```
* Make CompactSize patterns consistent with integer lengths
* Replace unwrap_err with asserting is_err
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Replace a panic with an assertion
* Make generic serialization use CompactSizeMessage
* Fix type inference and borrow-checker errors
* Doctest fixes
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add unused seed peers to the AddressBook
* Document a new `await`
We added an extra await on the AddressBook thread mutex.
Co-authored-by: teor <teor@riseup.net>
* Fix a typo
* Refactor names
* Return early from `limit_initial_peers`
* Add `proptest`s regressions
* Return `MetaAddr` instead of `None`
* Test if `zebra_network::init()` deadlocks
* Remove unneeded regressions
* Rename `TimestampCollector` to `AddressBookUpdater` (#2992)
* Rename `TimestampCollector` to `AddressBookUpdater`
* Update comments
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Move `all_peers` instead of copying them
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Make `Duration` a const
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use a timeout instead of measuring the elapsed time
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Copy `initial_peers` instead of moving them
* Refactor the position of `NewInitial` and `new_initial`
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Update `tower` to version `0.4.9`
Update to latest version to add support for Tokio version 1.
* Replace usage of `ServiceExt::ready_and`
It was deprecated in favor of `ServiceExt::ready`.
* Update Tokio dependency to version `1.13.0`
This will break the build because the code isn't ready for the update,
but future commits will fix the issues.
* Replace import of `tokio::stream::StreamExt`
Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.
* Use `IntervalStream` in `zebra-network`
In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.
* Use `IntervalStream` in `inventory_registry`
In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.
* Use `BroadcastStream` in `inventory_registry`
In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.
* Handle `Semaphore::acquire` error in `tower-batch`
Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.
* Handle `Semaphore::acquire` error in `zebrad` test
On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.
* Update some `zebra-network` dependencies
Use versions compatible with Tokio version 1.
* Upgrade Hyper to version 0.14
Use a version that supports Tokio version 1.
* Update `metrics` dependency to version 0.17
And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.
* Use `f64` as the histogram data type
`u64` isn't supported as the histogram data type in newer versions of
`metrics`.
* Update the initialization of the metrics component
Make it compatible with the new version of `metrics`.
* Simplify build version counter
Remove all constants and use the new `metrics::incement_counter!` macro.
* Change metrics output line to match on
The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.
* Update `sentry` to version 0.23.0
Use a version compatible with Tokio version 1.
* Remove usage of `TracingIntegration`
This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.
* Add sentry layer to tracing initialization
This seems like the replacement for `TracingIntegration`.
* Remove unnecessary conversion
Suggested by a Clippy lint.
* Update Cargo lock file
Apply all of the updates to dependencies.
* Ban duplicate tokio dependencies
Also ban git sources for tokio dependencies.
* Stop allowing sentry-tracing git repository in `deny.toml`
* Allow remaining duplicates after the tokio upgrade
* Use C: drive for CI build output on Windows
GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.
Co-authored-by: teor <teor@riseup.net>
* Use `prop_assert` instead of `assert`
Otherwise the test input isn't minimized.
* Split long string into a multi-line string
And add some newlines to try to improve readability.
* Fix referenced issue number
They had a typo in their number.
* Make peer services optional
It is unknown for initial peers.
* Fix `preserve_initial_untrusted_values` test
Now that it's optional, the services field can be written to if it was
previously empty.
* Fix formatting of property tests
Run rustfmt on them.
* Restore `TODO` comment
Make it easy to find planned improvements in the code.
Co-authored-by: teor <teor@riseup.net>
* Comment on how ordering is affected
Make it clear that missing services causes the peer to be chosen last.
Co-authored-by: teor <teor@riseup.net>
* Don't expect `services` to be available
Avoid a panic by using the compiler to help enforce the handling of the
case correctly.
* Panic if received gossiped address has no services
All received gossiped addresses have services. The only addresses that
don't have services configured are the initial seed addresses.
Co-authored-by: teor <teor@riseup.net>
* Add default deny.toml for "cargo deny check bans"
`cargo deny init`
* Delete unused "cargo deny" config entries
Also cleanup trailing whitespace.
* Deny duplicate crates and unexpected crate sources
Allow the current set of duplicates and sources,
with references to the tickets that will fix them.
* Check for duplicate dependencies in CI
Also check for:
- unexpected crate sources
- outdated Cargo.lock
(required for accurate duplicate and source checks)
* Revert CI name changes so required statuses pass
* Fix ticket for sentry-tracing
* Fix some links and linewraps
* Flesh out module-level rustdoc for mempool component
* Links for Downloads and Storage
* Tidy rustdoc links
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Avoid spurious test failures on busy machines
* Revert "Avoid spurious test failures on busy machines"
This reverts commit 4430a48ef3.
* Increase request delay for mock network service
This is necessary because some tests were having timeouts when running
on macOS for the CI. This meant that it took longer than expected for
the subject under test to end up sending the network request.
Co-authored-by: teor <teor@riseup.net>
* Improve documentation
* Incorporate text about Zebra from the last newsletter
* Organize README and user docs
* Add table of contents, organize heading levels
* Fix link
* capitalize list items
* fix table of contents
* format spacing issue
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Update zebra beta goals
* Update known issues
* Fix NU5 missing validation
* Update future work
* Add issue numbers to link descriptions for known issues
* Add some more known issues
* Sprout-on-Groth16 came in NU1 (Sapling)
* Add in 'Validation of JoinSplit proofs using BCTV14 verifier' under Sprout, for clarity
* Update current features
* Update future work
* Add link to github releases
* minor formatting typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Apply suggestions from code review
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Tidy links and notes for some mempool rustdoc
* Fix link
* Fix some links and linewraps
* minor spacing
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Reorder imports to follow convention
Place the imports from `std` at the top.
* Add transaction errors for double spends
Add a variant for each pool. They represent a double spend inside a
transaction.
* Add `check::spend_conflicts` implementation
Checks if a transaction has spend conflicts, i.e., if a transaction
spends a UTXO more than once or if it reveals a nullifier more than
once.
* Reject transactions with internal spend conflicts
The transaction verifier should reject transactions that spend the same
transparent UTXO or that reveal the same nullifier.
* Add transparent spend consensus rule
Add it to the documentation to help with understanding and auditing it.
Co-authored-by: teor <teor@riseup.net>
* Use different nullifiers by default
Don't use the same nullifier twice when mocking a
`sprout::JoinSplitData` because it will lead to an invalid transaction.
* Test transactions with repeated spend outpoints
Since that represents a spend conflict, they should be rejected.
* Test duplicate nullifiers in joinsplit
Check if a mock transaction with a joinsplit that reveals the same
nullifier twice is rejected.
* Test duplicate nullifiers across joinsplits
Check if a duplicate nullifier in two different joinsplits in the same
transaction is rejected.
* Test V4 transaction with duplicate Sapling spend
Check if a V4 transaction that has a duplicate Sapling spend is
rejected.
* Test V5 transaction with duplicate Sapling spend
Check if a V5 transaction that has a duplicate Sapling spend is
rejected.
* Test V5 transaction with duplicate Orchard actions
Check if a V5 transaction that has duplicate Orchard actions is rejected
by the transaction verifier.
Co-authored-by: teor <teor@riseup.net>
* Limit open inbound connections based on the config
* Log inbound connection errors at debug level
* Test inbound connection limits
* Use clone directly in function call argument lists
* Remove an outdated comment
* Update tests to use an unbounded channel rather than mem::forget
And rename some variables.
* Use a lower limit in a slow test and require that it is exceeded
* Rate-limit initial seed peer connections
* Revert "Rate-limit initial seed peer connections"
This reverts commit f779a1eb9e.
* Simplify logic
* Avoid cooperative async task starvation in the peer crawler and listener
If we don't yield in these loops, they can run for a long time before
tokio forces them to yield.
* Add test
* Check for task panics in initial peers test
* Remove duplicate code in rebase
Co-authored-by: teor <teor@riseup.net>
* Limit the number of outbound connections in the crawler
* Make zebra-network channel bounds depend on config.peerset_initial_target_size
* Bias Zebra towards outbound connections
And turn connection limits into `Config` methods.
* Downgrade some connection logs to debug
* Remove verbose or outdated fields in tracing logs
* Clarify connection limits
Includes:
- `fastmod OUTBOUND_PEER_BIAS_FRACTION OUTBOUND_PEER_BIAS_DENOMINATOR zebra*`
- clarify connection limit documentation
* Clarify inventory channel capacity
* Add zebra_network::initialize tests with limited numbers of peers
* Avoid cooperative async task starvation in the peer crawler and listener
If we don't yield in these loops, they can run for a long time before
tokio forces them to yield.
* Test the crawler with small connection limits
And use the multi-threaded runtime to avoid long hangs.
* Stop using the multi-threaded executor in tests where it's not needed
* Avoid starvation for every connection
Adds yields after inbound successes and initial peer connections.
* Add a crawler peer connection success test
* Add outbound connection limit tests
* Improve outbound tests
* Limit the size and age of the ZIP-401 rejected transaction ID list
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix bug in EvictionList; improve documentation
* Separate public and non-public parts of the documentation
* Fix tests
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix bug in EvictionList::len()
* Make EvictionList::len() mutable; prune the list inside it
* Limit the size of EvictedList::ordered_entries
* Increase eviction_list_time_mixed time constants to try to make it pass on MacOS
* Simplify logic by assuming refreshes will never happen
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Compiling fixes
* Remove MEMPOOL_SIZE and just rely on the ZIP-401 cost limit
Co-authored-by: teor <teor@riseup.net>
* ZIP-401 weighted random mempool eviction
* rename zcash.mempool.total_cost.bytes to zcash.mempool.cost.bytes
Co-authored-by: teor <teor@riseup.net>
* Remove duplicated lines
* Add cost() method to UnminedTx
Update serialization failure messages
* More docs quoting ZIP-401 rules
* Change mempool::Storage::new() to handle Copy-less HashMap, HashSet
* mempool: tidy cost types and evict_one()
* More consensus rule docs
* Refactor calculating mempool costs for Unmined transactions
* Add a note on asympotic performance of calculating weights of txs in mempool
* Bump test mempool / storage config to avoid weighted random cost limits
* Use mempool tx_cost_limit = u64::MAX for some tests
* Remove failing tests for now
* Allow(clippy::field-reassign-with-default) because of a move on a type that doesn't impl Copy
* Fix mistaken doctest formatting
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Increase test timeout for Windows builds
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Limit tx size
Zebra now limits the transaction size in the `zcash_deserialize()` method for
`Transaction`.
* Remove unused error variants (#2941)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Limit tx size
Zebra now limits the transaction size in the `zcash_deserialize()` method for
`Transaction`.
* Test the tx deserialization limit
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Wrap `Sleep` timer in a `Pin<Box<_>>`
The `Sleep` type doesn't implement `Unpin` in newer versions of Tokio.
* Wrap `Sleep` type in a `Pin<Box<_>>`
In newer Tokio versions the `Sleep` type doesn't implement `Unpin`, so
it needs to be manually pinned.
* Try simulating a chain growth
* Adjust the transaction expiry height
The mempool evicts expired transactions. When working with mocked data,
appending a new block typically clears the mempool because transactions become
expired. For this reason, the expiry height of each transactions is adjusted so
that it is greater than the new chain tip's height.
* Refactor the code so that it works with `VerifiedUnminedTx`
* Fix a typo
* Fix clippy warnings
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
There are a lot of these messages when Zebra starts up.
They might be slowing down CI and causing timeouts.
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Replace some unit tuples with named unit structs
This helps distinguish generic channels and make them type-safe.
Also tidy imports and documentation in `peer_set::set`.
* Link to the tower balance crate from docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
dependabot[bot]
teor
Marek
Deirdre Connolly
Alfredo Garcia
Janito Vaqueiro Ferreira Filho
Pili Guerra
Conrado Gouvea