Commit Graph

312 Commits

Author SHA1 Message Date
Jane Lusby e105b4f6c5
properly document guarantee provided by zebra-network (#713)
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2020-07-22 14:38:00 -04:00
Henry de Valence 4a41c9254d network: avoid panic when shutting down cleanly.
When the connection sees the client_rx channel close it knows it will never get
any more requests, and it should terminate.  But instead of terminating, it
errored itself, and the method to error itself tries to pull all the
outstanding client requests from the channel in order to fail them before it
shuts down.  This results in reading from a closed channel, causing a panic.
Instead we return cleanly rather than failing (since we know there are no
outstanding requests, as the channel is closed).
2020-07-22 18:04:45 +10:00
Henry de Valence 0dc2d92ad8 network: ensure dropping a Client closes the connection.
This fixes a bug introduced when we added heartbeat support.  Recall that we
handle the Bitcoin connection state machine on a per-peer basis.  Each
connection has a task created from the `Connection` struct, and a `Client:
tower::Service` "frontend" that passes requests to it via a channel.  In the
`Connection` event loop, the connection checks whether the request channel has
been closed, indicating no further requests from the `Client`, in which case it
shuts itself down and cleans up resources.  This occurs when all of the senders
have been dropped.

However, this behavior broke when we introduced heartbeat support, because we
spawned an additional task to send heartbeat messages along the request
channel.  This meant that instead of having a single sender, dropped by the
`Client`, we have two senders, the `Client` and the "shadow client" task that
generates heartbeat messages.  This means that when the `Client` is dropped, we
still have a live sender and the connection is not closed.  To fix this, the
`Client` now uses a `oneshot` to shut down its corresponding heartbeat task.
This closes all senders.
2020-07-21 15:43:31 -07:00
teor b0cd920fad feature: Use the Heartwood protocol version in zebra-network 2020-07-21 10:46:07 -07:00
teor 1cb1f1c52e fix: Put the peer set config vars together 2020-07-21 12:20:48 -04:00
dependabot[bot] c8fe4b43d8 build(deps): bump indexmap from 1.4.0 to 1.5.0
Bumps [indexmap](https://github.com/bluss/indexmap) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/bluss/indexmap/releases)
- [Commits](https://github.com/bluss/indexmap/compare/1.4.0...1.5.0)

Signed-off-by: dependabot[bot] <support@github.com>
2020-07-21 12:19:01 -04:00
Alfredo Garcia fe2a468417
add favicon to generated docs (#681) 2020-07-17 16:45:29 -07:00
teor ab6d1f5ec8
fix: Use the default Zcash port in version messages (#661)
We don't provide our address yet, so the port should be ignored.

But let's use the correct port, to avoid carrying this bug forward into
working code.
2020-07-15 11:43:28 -07:00
Alfredo Garcia d8834b149a
Limit protocol messages size (#645)
* change body msg limit and test case
* accept body at the exact limit len
* test the edges of the limit value
2020-07-15 19:15:52 +10:00
Henry de Valence fcd2f43f39 network: add warning to connection handling code. 2020-07-09 11:15:06 -07:00
Henry de Valence 217c25ef07 network: propagate tracing Spans through peer connection 2020-07-09 11:15:06 -07:00
Dimitris Apostolou ba81d7d4c0 Fix typos 2020-07-07 11:13:49 -07:00
teor f999ec75e6 fix: Remove a non-standard unicode character in a comment 2020-07-01 16:03:14 -04:00
Deirdre Connolly 05316dee21 Listen on 0.0.0.0, not 127.0.0.1
Turns out when your node faces the internet directly, it has to listen
to those addresses directly.
2020-06-19 03:46:09 -04:00
Henry de Valence 6cc1627a5d zebrad: apply serde(default) to config sections
Each subsection has to have `serde(default)` to get the behaviour we want
(delete all fields except the ones that have been changed); otherwise, we can
delete only entire sections.
2020-06-18 17:43:36 -04:00
Jane Lusby df18ac72c5 fix sharedpeererror to propagate tracing context 2020-06-17 14:38:26 -07:00
Jane Lusby 685bdaf2df don't require absense of cancel handles
Prior to this change, we required that services that are canceled do not
have a cancel handle in the `cancel_handles` list, based on the
assumption that the handle must have been removed in the process of
canceling this service.

This doesn't holding up though, because it is currently possible for us
to have the same peer connect to us multiple times, the second connect
removes the cancel handle of the original connect and inserts it's own
cancel handle in its place. In this scenario, when the first service is
polled for readiness it will see that it has been canceled and go to
clean itself up, but when it asserts that it doesn't have a cancel
handle it will see the cancel handle of the second connect event, which
uses the same key as the first connect, and fail its debug assertion.

This change removes that debug assert on the assumption that it is okay
for a peer to connect multiple times consecutively, and that the correct
behavior in that case is to just cancel the first connection and
continue as normal.
2020-06-16 13:42:31 -07:00
Jane Lusby 4b9e4520ce
cleanup API for arc based error type (#469)
Co-authored-by: Jane Lusby <jane@zfnd.org>
2020-06-12 11:29:42 -07:00
George Tankersley d8b3db5679 Use new seeder address for yolo.money 2020-06-10 21:49:25 -04:00
George Tankersley 6606bcaa62 Update list of DNS seeders
This adds the Foundation's new seeders and removes Simon's defunct one.
2020-06-10 20:56:31 -04:00
Jane Lusby 431f194c0f
propagate errors out of zebra_network::init (#435)
Prior to this change, the service returned by `zebra_network::init` would spawn background tasks that could silently fail, causing unexpected errors in the zebra_network service.

This change modifies the `PeerSet` that backs `zebra_network::init` to store all of the `JoinHandle`s for each background task it depends on. The `PeerSet` then checks this set of futures to see if any of them have exited with an error or a panic, and if they have it returns the error as part of `poll_ready`.
2020-06-09 12:24:28 -07:00
Jane Lusby 9f802cd8dd Wrap Transaction in Arc 2020-06-06 18:13:17 -04:00
Jane Lusby 9bcda0f9c7 Wrap Blocks in Arc throughout codebase 2020-06-05 00:36:55 -04:00
dependabot-preview[bot] 7a75522885 Bump indexmap from 1.3.2 to 1.4.0
Bumps [indexmap](https://github.com/bluss/indexmap) from 1.3.2 to 1.4.0.
- [Release notes](https://github.com/bluss/indexmap/releases)
- [Commits](https://github.com/bluss/indexmap/compare/1.3.2...1.4.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-06-01 15:38:00 -04:00
dependabot-preview[bot] 145d9a1835 Bump proptest from 0.9.6 to 0.10.0
Bumps [proptest](https://github.com/altsysrq/proptest) from 0.9.6 to 0.10.0.
- [Release notes](https://github.com/altsysrq/proptest/releases)
- [Changelog](https://github.com/AltSysrq/proptest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/altsysrq/proptest/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-05-29 15:06:40 -04:00
dependabot-preview[bot] e317b68b1d Bump proptest-derive from 0.1.2 to 0.2.0
Bumps [proptest-derive](https://github.com/AltSysrq/proptest) from 0.1.2 to 0.2.0.
- [Release notes](https://github.com/AltSysrq/proptest/releases)
- [Changelog](https://github.com/AltSysrq/proptest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/AltSysrq/proptest/compare/proptest-derive-0.1.2...proptest-derive-0.2.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-05-28 23:00:29 -04:00
Jane Lusby 4a2d2a359c
add cargo fmt to ci (#403)
* add cargo fmt to ci

* rebase on main

* switch to stable

Co-authored-by: Jane Lusby <jane@zfnd.org>
2020-05-27 19:12:25 -07:00
Jane Lusby 8c178c3ee4
fix panic in seed subcommand (#401)
Co-authored-by: Jane Lusby <jane@zfnd.org>

Prior to this change, the seed subcommand would consistently encounter a panic in one of the background tasks, but would continue running after the panic. This is indicative of two bugs. 

First, zebrad was not configured to treat panics as non recoverable and instead defaulted to the tokio defaults, which are to catch panics in tasks and return them via the join handle if available, or to print them if the join handle has been discarded. This is likely a poor fit for zebrad as an application, we do not need to maximize uptime or minimize the extent of an outage should one of our tasks / services start encountering panics. Ignoring a panic increases our risk of observing invalid state, causing all sorts of wild and bad bugs. To deal with this we've switched the default panic behavior from `unwind` to `abort`. This makes panics fail immediately and take down the entire application, regardless of where they occur, which is consistent with our treatment of misbehaving connections.

The second bug is the panic itself. This was triggered by a duplicate entry in the initial_peers set. To fix this we've switched the storage for the peers from a `Vec` to a `HashSet`, which has similar properties but guarantees uniqueness of its keys.
2020-05-27 17:40:12 -07:00
Jane Lusby 8276bed400 reinstate reject error variant 2020-05-27 15:42:29 -04:00
Jane Lusby 4dc307f2f3 fix last warnings 2020-05-27 15:42:29 -04:00
Jane Lusby b6b35364f3 cleanup warnings throughout codebase 2020-05-27 15:42:29 -04:00
George Tankersley df79fa75e0
Implement minimal version handshaking (#295)
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
2020-04-13 18:33:15 -04:00
Deirdre Connolly a5f4db7528 Move just the Network enum to -chain, keep everything else in -network 2020-03-12 22:02:17 -04:00
Deirdre Connolly 380d622b37 Fix imports 2020-03-12 22:02:17 -04:00
Deirdre Connolly b68e1e2d55 Move Network, Magic, and magics to zebra-chain 2020-03-12 22:02:17 -04:00
Deirdre Connolly 8c0b00109f Remove PeerError::DeadServer, unused, unneeded
Resolves #251
2020-03-12 16:23:08 -04:00
Henry de Valence ff3efd504c Add Zebra logo to all workspace crates.
Also add html_root_url attributes.
2020-02-26 21:25:35 -08:00
Henry de Valence 3ed75cb626 Tweak peer set metrics.
- Add a total peers metric to prevent races between measurements of
  ready/unready peers (which can cause the sum to be wrong).
- Add an outbound request counter.
2020-02-21 06:48:25 -05:00
Henry de Valence 94fe2c3b57 Increase the peerset request buffer size.
tower-buffer uses tokio's mpsc channels, not the futures-rs mpsc channels.
Unlike futures-rs mpsc channels, which have capacity n+m, where n is the buffer
size and m is the number of senders, tokio channels always have buffer size n.
This means that the buffer size is shared across all peer set handles.

Thanks to @hawkw for sharing details of the Tokio internals!
2020-02-21 06:48:25 -05:00
Henry de Valence 5f07a25b05 Shorten the default new_peer_interval to 60s.
This increases the frequency at which we crawl the network.
2020-02-21 06:48:25 -05:00
Henry de Valence 80e7ee6dae Add metrics for inbound and outbound messages. 2020-02-21 06:48:25 -05:00
Henry de Valence 8c938af579 Spawn tasks for handshake futures.
Previously, we relied on the owner of the handshake future to drive it to
completion.  This meant that there were cases where handshakes might never be
completed, just because nothing was actively polling them.
2020-02-21 06:48:25 -05:00
Henry de Valence 43b2d35dda Crawl for more peers when we exhaust candidates. 2020-02-21 06:48:25 -05:00
Henry de Valence afa2c2347f fmt 2020-02-21 06:48:25 -05:00
Henry de Valence 00edcae0c2 Add metrics for the crawler and candidate set. 2020-02-14 20:14:05 -05:00
Henry de Valence 75d3d44fb3 Metrics MVP: add two metrics and export them to Prometheus.
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2020-02-14 20:14:05 -05:00
Henry de Valence 8000f888fd Connect to multiple peers concurrently.
The previous outbound peer connection logic got requests to connect to new
peers and processed them one at a time, making single connection attempts
and retrying if the connection attempt failed.  This was quite slow, because
many connections fail, and we have to wait for timeouts.  Instead, this logic
connects to new peers concurrently (up to 50 at a time).
2020-02-14 18:23:41 -05:00
Henry de Valence 7049f9d891 Add a FindBlocks request to get initial block hashes.
Bitcoin does this either with `getblocks` (returns up to 500 following block
hashes) or `getheaders` (returns up to 2000 following block headers, not
just hashes).  However, Bitcoin headers are much smaller than Zcash
headers, which contain a giant Equihash solution block, and many Zcash
blocks don't have many transactions in them, so the block header is
often similarly sized to the block itself.  Because we're
aiming to have a highly parallel network layer, it seems better to use
`getblocks` to implement `FindBlocks` (which is necessarily sequential)
and parallelize the processing of the block downloads.
2020-02-14 18:23:41 -05:00
Henry de Valence 47cafc630f Remove version fields from GetBlocks, GetHeaders.
These are instead set by the negotiated version.
2020-02-14 18:23:41 -05:00
Henry de Valence abcc0a6773 Add basic retry policies to zebra-network.
This should be removed when https://github.com/tower-rs/tower/pull/414 lands
but is good enough for our purposes for now.
2020-02-11 15:23:19 -05:00
Henry de Valence befdb46dc3 Clean some warnings in the Bitcoin codec.
This doesn't clean the warnings about unused items in the builder, since
those are unused for a reason (the implementation that should use them
is missing).
2020-02-10 09:03:56 -08:00
Henry de Valence 2082672b3c Remove Response::Error.
Error handling is already handled by Result; we don't need an "inner"
error variant duplicating the outer one.
2020-02-10 09:03:56 -08:00
Henry de Valence 29f901add3 Rename Response::Ok to Response::Nil.
This is a better name because it signals "no data in response" rather
than "Ok", which is semantically mixed with `Ok/Err` of `Result`.
2020-02-10 09:03:56 -08:00
Henry de Valence 5929e05e52 Remove `PushPeers` and ignore unsolicited `addr` messages.
PushPeers is more complicated to thread into the rest of our
architecture (we would need to establish a data path connecting our
service handling inbound requests to the network layer's auto-crawler),
and since we crawl the network automatically anyways, we don't actually
need to accept them in order to get updated address information.

The only possible problem with this approach is that zcashd refuses to
answer multiple address requests from the same connection, ostensibly
for fingerprinting prevention (although it's totally happy to give
exactly the same information, as long as you hang up and reconnect
first, lol).  It's unclear how this will interact with our design -- on
the one hand, it could mean that we don't get new addr information when
we ask, but on the other hand, we may have enough churn in our
connection pool that this isn't a problem anyways.
2020-02-10 09:03:56 -08:00
Henry de Valence 2c0f48b587 Refactor connection logic and try a block request.
Attempting to implement requests for block data revealed a problem with
the previous connection logic.  Block data is requested by sending a
`getdata` message with hashes of the requested blocks; the peer responds
with a sequence of `block` messages with the blocks themselves.

However, this wasn't possible to handle with the previous connection
logic, which could only convert a single Bitcoin message into a
Response.  Instead, we factor out the message handling logic into a
Handler, which can statefully accumulate arbitrary data into a Response
and signal completion.  This is still pretty ugly but it does work.

As a side effect, the HeartbeatNonceMismatch error is removed; because
the Handler now tries to process messages until it comes to a Response,
it just ignores mismatched nonces (and will eventually time out).

The previous Mempool and Transaction requests were removed but could be
re-added in a different form later.  Also, the `Get` prefixes are
removed from `Request` to tidy the name.
2020-02-10 09:03:56 -08:00
Henry de Valence 972d16518f Make ZcashSerialize infallible mod its Writer.
Closes #158.

As discussed on the issue, this makes it possible to safely serialize
data into hashes, and encourages serializable data to make illegal
states unrepresentable.
2020-02-05 19:48:43 -05:00
Henry de Valence b0f61c4dd2 Remove outdated comment (we use tokio codecs now) 2020-02-05 19:44:35 -05:00
Henry de Valence ab94acf7da fmt 2020-02-05 19:44:35 -05:00
Henry de Valence eeb4a2470b Remove version fields from Block, Tx messages.
These are included in the Block, Transaction objects themselves, so the
previous code ended up trying to deserialize two version fields per
object.

Closes #226.
2020-02-05 19:44:35 -05:00
Henry de Valence 51c744b1ae Update network version number. 2020-02-05 14:06:35 -08:00
Henry de Valence 8d58dd804f Note that tracing causes clippy false positives
Thanks @hawkw for pointing this out.
2020-02-05 12:42:32 -08:00
Henry de Valence f04f4f0b98 Apply clippy fixes 2020-02-05 12:42:32 -08:00
dependabot-preview[bot] 979cf7ac6d Bump indexmap from 1.3.1 to 1.3.2
Bumps [indexmap](https://github.com/bluss/indexmap) from 1.3.1 to 1.3.2.
- [Release notes](https://github.com/bluss/indexmap/releases)
- [Commits](https://github.com/bluss/indexmap/compare/1.3.1...1.3.2)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-02-05 14:14:51 -05:00
Henry de Valence b24f53f4a1 Add From impls for InventoryHash 2020-02-04 17:48:36 -08:00
Deirdre Connolly 6d3d4c4f64 s/GetData/NotFound/ in read_notfound 2020-02-04 18:04:53 -05:00
Deirdre Connolly 1ca55846eb Little test to exercise sha256dWriter::flush() 2020-02-04 18:04:53 -05:00
Deirdre Connolly beb72080cb Delete out of date comment on incomplete Message variants 2020-01-30 13:39:13 -05:00
Deirdre Connolly 53a7af82a0 Add back a missing quotemark
Co-Authored-By: Henry de Valence <hdevalence@hdevalence.ca>
2020-01-28 03:48:23 -05:00
Deirdre Connolly 71d5571e39 Add roundtrip proptest for LockTime serialization/deserialization
Relates to #150
2020-01-28 03:48:23 -05:00
Deirdre Connolly d8ebeea08c Add proptest regressions file 2020-01-28 03:48:23 -05:00
Deirdre Connolly c2411f4315 Add a little proptest around Magic's Debug impl 2020-01-28 03:48:23 -05:00
Deirdre Connolly 9709b54c57 Fix proptest-regressions locations after reorganizations of files 2020-01-28 03:48:23 -05:00
Henry de Valence bea7b195ce Remove vestigial should_be_private module. 2020-01-16 19:27:08 -05:00
Henry de Valence 7cc44f4fa9 Move server.rs to connection.rs and change imports. 2020-01-16 13:20:03 -05:00
Henry de Valence 77ad61331c Rename `peer::Server` to `peer::Connection`.
This doesn't change the file path or edit imports so that the diff is easier to review.
2020-01-16 13:20:03 -05:00
Deirdre Connolly a3b09c3690
Target crates.io 0.3 versions of tower and tower-load (#172) 2020-01-14 17:18:14 -05:00
Henry de Valence 92ddf0542f Provide impl Zcash[De]Serialize for Vec<T: Zcash[De]Serialize>.
This replaces the read_list function and makes the code significantly cleaner.

The only downside is that it loses exact preallocation, but this is probably not a big deal.
2019-12-31 02:46:39 -05:00
Henry de Valence 2965187b91 Upgrade tokio, futures, hyper to released versions. 2019-12-13 17:42:15 -05:00
Deirdre Connolly 82e246d87b
Merge pull request #135 from ZcashFoundation/130
On receipt of a Filter(Load|Add|Clear) message, disconnect from peer
2019-12-05 14:06:05 -05:00
Deirdre Connolly d559a3fa52 Return an error result if the filterload msg size is out of bounds 2019-12-05 13:48:42 -05:00
Deirdre Connolly cbd95dfc1f Pass body_len to read_filterload instead of calculating it ourself
This lets us still be generic over the reader trait and we aren't recomputing  again.

Co-Authored-By: Henry de Valence <hdevalence@hdevalence.ca>
2019-12-05 13:48:42 -05:00
Deirdre Connolly 3fbfffeb44 Remove superfluous zcash_(de)serialize impls 2019-12-05 13:48:42 -05:00
Deirdre Connolly 9a0c2198aa Parse out exact/max possible lengths for filterload filters
Add some unit tests that make sure we obey max values.
2019-12-05 13:48:42 -05:00
Deirdre Connolly 78f0b8aab0 Remove NODE_BLOOM service bit
We do not support bloom filters.
2019-12-05 13:48:42 -05:00
Deirdre Connolly 3c26092b01 Read max of 520 bytes for data fields when parsing filteradd messages 2019-12-05 13:48:42 -05:00
Deirdre Connolly 1b8b4d0fac Encode and decode Filter* messages
Also remove stubs related to MerkleBlock and the catchall case for unknown types since we
are finally matching all variants of the Message enum.
2019-12-05 13:48:42 -05:00
Deirdre Connolly 72def27b78 Add FilterLoad, FilterAdd, FilterClear message variants
Also remove MerkleBlock stub, we are no longer supporting it (zcashd doesn't either).
2019-12-05 13:48:42 -05:00
Deirdre Connolly f5aa5f3794 Add Tweak and Filter types 2019-12-05 13:48:42 -05:00
Henry de Valence 36cd6d6e06 cargo fmt 2019-11-27 23:53:36 -05:00
Henry de Valence f58aaac1ae Privately re-export ErrorSlot, ClientRequest in peer.
This means that all sub-modules of `peer` can import everything they need from
the `peer` module itself, without having to be aware of the internal structure
of their sibling modules.
2019-11-27 23:53:36 -05:00
Henry de Valence ad6525574b Rename PeerConnector -> peer::Connector 2019-11-27 23:53:36 -05:00
Henry de Valence 778e49b127 Rename PeerHandshake -> peer::Handshake 2019-11-27 23:53:36 -05:00
Henry de Valence 9ff0fd90dc Rename ServerState -> State.
There's no need to write `Server` every time because it's only used inside of
the server code, and when the handshake service constructs a Server.
2019-11-27 23:53:36 -05:00
Henry de Valence d1b3e8fe6b Rename PeerServer -> peer::Server 2019-11-27 23:53:36 -05:00
Henry de Valence 77191e62f6 Remove outdated fixup note. 2019-11-27 23:53:36 -05:00
Henry de Valence da78603d3a Rename `PeerClient` to `peer::Client`. 2019-11-27 23:53:36 -05:00
Henry de Valence 4fbc8270a2 Move PeerSet initialization into a submodule. 2019-11-27 05:06:01 -05:00
Henry de Valence 6db852fab2 Refactor protocol into internal, external modules.
This commit just moves things around and patches import paths.
2019-11-27 05:06:01 -05:00
Deirdre Connolly 8a9a5ba29b Revert "Add some simple proptests using the Arbitray trait on Requests and Responses, gated to test only"
This reverts commit 5a123acf56.
2019-11-26 19:35:49 -05:00
Deirdre Connolly 6f52fc7773 Add 'Other' opcode matching when parsing Reject messages 2019-11-26 19:35:49 -05:00