* Move version into a ConnectionInfo struct
* Add negotiated version to ConnectionInfo
Part of this change was generated using:
```
fastmod --fixed-strings ".version(" ".remote_version(" zebra-network
```
* Add the peer address to ConnectionInfo, add ConnectionInfo to Connection
* Return a Client instance from connect_isolated_* functions
This allows library users to access client ConnectionInfo.
* Add and improve debug formatting
* Add peer services and user agent to ConnectionInfo
* Export the Client type, and fix up a zebrad test
* Export types used by the public API
* Split VersionMessage into its own struct
* Use VersionMessage in ConnectionInfo
* Add a public API test for ConnectionInfo
* Wrap ConnectionInfo in an Arc
* Fix some doc links
* Increase the full sync timeout to 28 hours
And put the timeout constant with the rest of them.
* Update developer documentation for the new constant location
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* upgrade config files test
* update cache data to generic in config files
* replace config file v14 with v15
* add test to check if we have the current config stored
* Explain how to fix the test failure
Co-authored-by: teor <teor@riseup.net>
* adds non-blocking writer for tracing subscriber
* use non_blocking writer for the fmt::Layer with the tokio-console feature as well
* adds doc comment to _guard field
* adds acceptance test
* update filter_handle type to use NonBlocking
* adds more detail on lossy non-blocking writer and sets tracing.filter to "trace" in acceptance test
* drops ZebradApp before process::exit(1) in the event of a FrameworkError
* reduces buffered lines limit to 8000
* adds tracing.buffer_limit config and some comments
* update acceptance.rs
* fix acceptance test
* fixes ambigious phrasing in comment
* updates zebrad/src/application.rs
* Find out what the join error is in the GitHub runner tests
* updates acceptance test to use recv_timeout instead of always waiting 10 seconds, removes unnecessary echo command, and reduces # of rpc requests to 500
* see if sleeping for a few seconds before exiting helps the macOS test pass
* Expand exit sleep docs
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* bump prost, tonic and tonic-build
* add protoc as a dependency step in the CI
* bump console-subscriber
* add protoc to `build-crates-individually`
* add protoc to docs build
* install protoc in lint.yml
* change protoc installation location in lint.yml
* add protoc to `Check Cargo.lock is up to date`
* ci(build): keep protoc pinned to the same major version
* ci(build): avoid rate limiting with `arduino/setup-protoc@v1`
* cargo upgrade --workspace console-subscriber
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Co-authored-by: teor <teor@riseup.net>
* add grpc mempool test research
* add a config flag for mempool injection of transactions in test
* Only copy the inner state directory in the send transactions test
* Preload Zcash parameters in some transaction verification tests
* Add a block and transaction Hash method to convert from display order bytes
* Update test coverage docs
* Add debugging output for mempool transaction verification
* Test fetching sent mempool transactions using gRPC
* Add extra log checks to the send transaction test
* Wait for zebrad mempool activation before running gRPC tests
* Update send transaction test for lightwalletd not returning mempool transactions
* Check zebrad logs instead of disabled lightwalletd gRPCs
* Add a debug option that makes RPCs pretend the sync is finished
* Remove an unused debug option
* Remove unused test code and downgrade some logs
* Fix test log checks
* Fix some rustdoc warnings
* Fix a compilation error due to new function arguments
* Make zebrad sync timeouts consistent and remove outdated code
* Document how to increase temporary directory space for tests
* Stop checking for a log that doesn't always happen
* Remove some commented-out code
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Update a comment about run time
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Add new config to new tests from the `main` branch
* Add transactions to the list, rather than replacing the list with each new block
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Add a config for increasing the number of RPC threads
* Add unit tests for parallel RPC servers
* Refactor tests to remove duplicate code
* Update the README
* Actually use parallel threads in some RPC tests
* Rename a function to prepare_block_header_and_transaction_data_batch()
* Fix formatting of test command timeouts and child process output
* Put some #[cfg()]s in the standard Rust location
* Update some test timings
* Allow code timers to be ignored
* adds start as default subcommand for zebrad
* moves EntryPoint to submodule and adds a test
* moves all start tests to config_test to avoid listener conflicts
* Update zebrad/src/application/entry_point.rs docs
* Revert "moves all start tests to config_test to avoid listener conflicts"
This reverts commit 61ce46f5a1.
* Update based on test API changes from another PR
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Handle test failure regexes using Result::Err, rather than panicking
* Add output logs to test context, and add tests for that
* Let empty test child logs be read again (and produce empty output)
* Ignore missing test children when killing with ignore_exited
* Fix a clippy lint
* Rename `line` to `line_result` for clarity
* Revert a redundant context_from() on kill()
* Only ignore "no such process" kill() errors in sync_until() tests
* Log the command timeout when an acceptance test fails
* fix clippy
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Show the arguments of acceptance test functions in the logs
* Show all the logs in the "Run tests" jobs
* Document expected "broken pipe" error from `tee`
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Increase full sync timeout to 24 hours
Expected sync time is ~21 hours as of August 2022.
* Split final checkpoint job into two smaller jobs to avoid timeouts
Also make regexes easier to read.
* Fix a job name typo
* update timeout
* update the doc comment
* Increase test timeouts for Zebra update syncs
* Stop failing the 1740k job if the cached state is after block 1740k
Co-authored-by: teor <teor@riseup.net>
* truncate the number of transactions in send transaction test
* Limit send transaction test to RPC queue length
Co-authored-by: teor <teor@riseup.net>
* update zebra to 1.0.0-beta.13
* add changelog
* update the release date
* Update changelog for cryptographic batch PRs
* Add `bitvec` to the changelog
* Update CHANGELOG.md
* Add zebra-network fix and tokio version bump
* Recommend that users update their rust compiler
* Update disk and network usage for recent large blocks
* Fix spacing
* Add disk and network usage update to README
* apply size suggestions
Co-authored-by: teor <teor@riseup.net>
* add known performance issues
* update changelog
* change release date
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make code execution time logs shorter
* Do ZK parameter preloads in the lightwalletd tests that need them
* Try to re-launch `lightwalletd` when it hangs during sync tests
* Increase full sync timeout
* Clear the `zebrad` logs during `lightwalletd` tests, to avoid logging deadlocks
* Actually clear more than one line of logs
* Check zebrad and lightwalletd output in parallel threads, while waiting for zebrad
* Check zebrad and lightwalletd output in parallel threads, while waiting for lightwalletd
* Improve test logging
* Fix a log typo
* Only wait for lightwalletd once, because its logs stop after the initial sync
* Look for cached state disks for this commit and branch first
* Only copy the state once in the send transactions test
* Wait longer for lightwalletd gRPC server startup
* Add some function docs
* cargo fmt --all
* Fix clippy::let_and_return
* Increase lightwalletd test timeouts for zebrad slowness
* Add a `zebrad_update_sync()` test, that update syncs Zebra without lightwalletd
* Run the zebrad-update-sync test in CI
* Add extra zebrad time to workaround lightwalletd bugs
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography
* Improve anchor validation debugging and error messages
* Work around a test data bug, and save some CPU
* Remove redundant checks for empty shielded data
* Skip generating unused interstitial treestates
* Do disk fetches and quick checks, then CPU-heavy cryptography
* Wrap HistoryTree in an Arc in the state
* Run CPU-intensive chain validation and updates in parallel rayon threads
* Refactor to prepare for parallel tree root calculations
* Run finalized state note commitment tree root updates in parallel rayon threads
* Update finalized state note commitment trees using parallel rayon threads
* Fix a comment typo and add a TODO
* Split sprout treestate fetch into its own function
* Move parallel note commitment trees to zebra-chain
* Re-calculate the tree roots in the same parallel batches
* Do non-finalized note commitment tree updates in parallel threads
* Update comments about note commitment tree rebuilds
* Do post-fork tree updates in parallel threads
* Add a TODO for parallel tree updates in tests
* Fix broken intra-doc links
* Clarify documentation for sprout treestates
* Spawn large database reads into blocking tokio threads
* Concurrently read all blocks, headers, and transactions from disk
* Run zebra-state transaction deserialization on a rayon thread
* Update comments about note commitment tree rebuilds
* Add info-level logs to time note commitment tree rebuilds
* Log the number of rebuilt blocks after a fork, and the time per block
* Move humantime formats to zebra-chain
* Use human-friendly time formatting
* Only fetch block headers from the database to answer headers requests
* Move writing to the database to a blocking thread
* Add blocking threads to tests that need them
* Remove mempool downloader requirement for a populated state
And stop populating states that don't need it,
so we can use tokio::time::pause() in those tests.
* Improve debugging for an intermittent test failure
* Try to avoid a race condition populating the mempool in tests
* try to do deserialization of transaction in a rayon thread
* Try tokio::task::block_in_place instead
* fix tests
* add deserialize block into rayon pool
* fill some docs
Co-authored-by: teor <teor@riseup.net>
* Initialize the rayon threadpool with a new config for CPU-bound threads
* Verify proofs and signatures on the rayon thread pool
* Only spawn one concurrent batch per verifier, for now
* Allow tower-batch to queue multiple batches
* Fix up a potentially incorrect comment
* Rename some variables for concurrent batches
* Spawn multiple batches concurrently, without any limits
* Simplify batch worker loop using OptionFuture
* Clear pending batches once they finish
* Stop accepting new items when we're at the concurrent batch limit
* Fail queued requests on drop
* Move pending_items and the batch timer into the worker struct
* Add worker fields to batch trace logs
* Run docker tests on PR series
* During full verification, process 20 blocks concurrently
* Remove an outdated comment about yielding to other tasks
* Return the maximum checkpoint height from the chain verifier
* Return the verified block height from the sync downloader
* Track the verified height in the syncer
* Use a lower concurrency limit during full verification
* Get the tip from the state before the first verified block
* Limit the number of submitted download and verify blocks in a batch
* Adjust lookahead limits when transitioning to full verification
* Keep unused extra hashes and submit them to the downloader later
* Remove redundant verified_height and state_tip()
* Split the checkpoint and full verify concurrency configs
* Decrease full verification concurrency to 5 blocks
10 concurrent blocks causes 3 minute stalls on some blocks on my machine.
(And it has about 4x as many cores as a standard machine.)
* cargo +stable fmt --all
* Remove a log that's verbose with smaller lookahead limits
* Apply the full verify concurrency limit to the inbound service
* Add a summary of the config changes to the CHANGELOG
* Increase the default full verify concurrency limit to 30
* Checkout zebra in each job to avoid warnings
But put TODOs where we might be able to skip checkouts
* Split log following into sprout checkpoints, sapling/orchard checkpoints, and full validation
* Make job IDs shorter
* Use /dev/stderr because docker doesn't have a tty
* remove pipefail
* Revert "remove pipefail"
This reverts commit a7ee37bebdc107a4215e7dd307b189d925969234.
* Make tee ignore errors writing to a grep pipe
* Avoid launching multiple docker instances for duplicate jobs
* Ignore broken pipe error messages and statuses
* fix(ci): docker wait not finding container
We had this issue before, I can't recall if this was a parsing error between GitHub Actions and gcloud `--command` parsing, but we had to change this into two pieces.
This implementation keeps it how we did it before 9b9578c999/.github/workflows/test.yml (L235-L243)
* docs: remove pending TODO
We can't remove `actions/checkout` nor set `create_credentials_file` to `false` as next steps won't be able to authenticate to GCP.
We can surely remove `actions/checkout` and leave `create_credentials_file` as `true`, but this will raise a warning on each step, and there's no benefit of doing so.
* Show `docker wait` and `gcloud ssh` output
* If `docker wait` fails, get the exit code using `docker inspect`
* Make full sync tests go all the way to the tip
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Update Zebra to 1.0.0-beta.12
* Simplify tower-* versioning by updating both to 0.2.27
* Simplify version search and replace by removing an old version from the docs
* Update Cargo.lock
* Add draft changelog as of PR #4693
* Update CHANGELOG to PR #4680
* Document configs that need compile-time features
* Document features in zebrad/src/lib.rs
* Link to the feature list from the README
* Remove some outdated README info
* Move some detailed README info to the `zebrad` crate docs
* Improve time logging using humantime
* Only log full seconds, ignore the fractional part
* Move humantime_seconds to tracing::fmt
* Move the progress task to its own module
* Add missing humantime dependency
* Log the network upgrade in progress logs
* Log when Zebra verifies the final checkpoint
* Warn the user when Zebra cannot parse `zebrad.toml`
* Test that Zebra warns the user when it cannot parse `zebrad.toml`
* Fix up a mistaken merge change
* Suggest how to fix `zebrad.toml` when Zebra cannot parse it
Co-authored-by: teor <teor@riseup.net>
* change `initial_mainnet_peers` and `initial_testnet_peers` type to `IndexSet`
* add tests for zebra config files
* add serde feature to indexmap
* remove async
* update config
* fix `stored_config_path()`
* skip tests if config is not found
* improve error
* use CARGO_MANIFEST_DIR
* remove `stored_config_is_newest` test
* move `stored_config_works` test to the end of `valid_generated_config_test`
* space
* Decrease the default lookahead limit to 400
* Increase the block verification timeout to 10 minutes
* Halve the default concurrent downloads config
* Try to run the spawned download task before queueing the next download
* Allow verification to be cancelled if the verifier is busy
* delete old database directories
* check if state directory exists
* skip deleting when ephemeral
* split `check_and_delete_old_databases`
* move `check_and_delete_old_databases` to state
* spawn `check_and_delete_old_databases`
* simplity a bit
* fix(state): only delete old database directories inside the cache directory (#4631)
* Add function comments, tweak log
* Simplify version parsing
* Use spawn_blocking to launch the task on a separate thread, do the cleanup last
* Abort the cleanup task when Zebra exits
* Split directory deletion into its own function, handle ownership
* Rename cache_dir to state_dir
* If an outdated state directory is outside the cache directory, don't delete it
* Minimise diffs
* add test
* fix typos
Co-authored-by: teor <teor@riseup.net>
* add `canonicalize` to test regex
* add another match to test
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Disable the flamegraph feature by default at compile time
* Disable the journald feature by default at compile time
* Also disable inferno dependency, and rearrange features
* Disable the prometheus feature by default at compile time
* Disable the tracing filter reload feature by default at compile time
* Disable tests when corresponding features are disabled
* Add compile-time tracing features to user docs
* Add compile-time features to the metrics user docs
* Document diagnostics as part of the start command tasks and services
* breaking(diagnostics): rename "enable-sentry" feature to "sentry" (#4623)
* Also skip conflict tests when those ports are disabled
* breaking(diagnostics): rename "enable-sentry" feature to "sentry"
This is mostly:
```sh
fastmod enable-sentry sentry
```
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Always activate tokio/tracing feature
And always build tests with all tokio features.
* Refactor tracing-subscriber init to simplify it
* Add the tokio-console feature and dependencies
* Add optional tokio-console support, and log the installed tracing layers at info level
Uses a tracing Registry for tokio-console, and a fmt::Subscriber otherwise.
* Add some TODOs based on tracing-subscriber features
* Fix up some spans
* Add a TODO for fixing a log filter bug in tokio-console mode
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fa.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
* Fix some warnings produced by `cargo doc`
* Fix some rustdoc warnings
* Fix some warnings
* Refactor some changes
* Fix some rustdoc warnings
* Fix some rustdoc warnings
* Resolve various TODOs
Co-authored-by: teor <teor@riseup.net>
* Fix some unresolved links
* Allow links to private items
* Fix some unresolved links
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* use `humantime_serde` for config durations
* move debug config option to the bottom
* fix deserialization
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move peer address validation into its own module
* Add a network parameter to AddressBook and some MetaAddr methods
* Reject invalid initial peers, and connect to them in preferred order
* Reject Flux/ZelCash and misconfigured Zcash peers
* Prefer canonical Zcash ports
* Make peer_preference into a struct method
* Prefer peer addresses with canonical ports for outbound connections
* Also ignore the Zcash regtest port
* Document where field and variant order is required for correctness
* Use the correct peer count
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* increase lightwalletd timeout
* switch back to aditya's fork
* manually point to new aditya's lightwalletd image
* disable sync_one_checkpoint_testnet test
* disable restart_stop_at_height in testnet
* rever to 'latest' lightwalletd image
dependabot[bot]
teor
Alfredo Garcia
Arya
Marek
Conrado Gouvea