* Refactor out try_to_sync_once()
* Add outer timeouts for obtaining and extending tips
* Refactor out request_genesis_once()
* Wrap genesis download once in a timeout
* Increase the genesis timeout to avoid denial of service from old nodes
* Add an outer timeout to mempool crawls
* Add an outer timeout to mempool download/verify
* Remove threaded mutex blocking from the inbound service
* Explain why inbound readiness never hangs
* Fix whitespace that cargo fmt doesn't
* Avoid hangs by always resetting the past lookahead limit flag
* Document block-specific and syncer-wide errors
* Update zebrad/src/components/sync.rs
Co-authored-by: Marek <mail@marek.onl>
* Use correct condition for log messages
Co-authored-by: Marek <mail@marek.onl>
* Keep lookahead reset metric
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: Marek <mail@marek.onl>
* Add a progress bar config that is disabled unless the feature is on
* Simplify the default config
* Enable the progress bar feature by default, but require the config
* Rename progress bars config to avoid merge conflicts
* Use a log file when the progress bar is activated
* Document how to configure progress bars
* Handle log files in config_tests and check config path
* Fix doc link
* Fix path check
* Fix config log matching
* Fix clippy warning
* Add tracing to config tests
* It's zebrad not zebra
* cargo fmt --all
* Update release for config file changes
* Fix config test failures
* Allow printing to stdout in a method
* Adds most_recent_by_ip field to address book
* adds test
* Apply suggestions from code review
* fixes lint
* Updates most_recent_by_ip in .take()
Updates should_update_most_recent_by_ip() and has_active_peer_with_ip to check last_attempt and last_failure times
Renames has_active_peer_with_ip
* Documents that Zebra will not initiate more than 1 outbound connections per IP
* Fixes is_ready_for_connection_attempt_with_ip
Adds test coverage for AttemptPending and Failed
Fixes new_outbound_peer_connections_are_rate_limited proptest
* Applies suggestions from code review.
* Applies suggestions from code review
* Always return true from `is_ready_for_connection_attempt_with_ip` if max_connections_per_ip != 0
* Update max_connections_per_ip config docs
* Warn about invalid config fields and use default values
* Ignores last_attempt and last_failure in is_ready_for_connection_attempt_with_ip
updates test
* Only update most_recent_by_ip if update.last_conn_state is responded.
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* fixes lint
* Update zebra-network/src/address_book.rs
Co-authored-by: teor <teor@riseup.net>
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix Rust syntax
* Fix whitespace
---------
Co-authored-by: teor <teor@riseup.net>
* Remove duplicate "running" message, send log file opening logs to stderr
* Add missing block progress bar desc, make them consistent
* Put progress bars in a specific order in each section
* Actually make block progress bar desc consistent
* Make progress bar order even more consistent
* Fix fork blocks plural for 1 block
* Use the correct number of chain fork bars
* Disable confusing partial work display
* Add struct field category comments
* Silence a verbose inventory log
* Update license description in README for MIT-only crates
* Draft changelog with trivial issues
* Remove trivial issues
* Update changelog entries as of commit 2a31972 and PR #7103
* Update mainnet and testnet checkpoints as of 2023-06-30
* chore: Release
* Estimate release height for Zebra v1.0.1
Block height 2139118 at 2023-06-30 01:55:38 UTC
Release is likely to be 2023-07-01
2139118 + 1152 * 3 = 2142574
Then round up to the nearest 1000.
* Stop busy-waiting in a Future for 45 seconds every minute
* Use the correct elapsed time calculation
* Add some TODOs for making the structure of the loop and wait times clearer
* print a Zebra logo and some text in progress bar mode
* add network to printed line, add heart to logo
* print logo and message regardless of progress-bar; document how logo was generated
* De-duplicate app_version and user_agent code, rename to build_version
* Make RPC testnet flag forward-compatible with additional testnets
* Fix RPC tests with new argument
* Use "modified" rather than "dirty" for uncommitted changes in build metadata
* Split the vergen version into its own function
* Add a missing timeout to outbound TCP connections
* Move inbound handshakes into their own function, replacing the manual span
* Delete a useless manual span in zebra_network::config
* Add an extra timeout to the spawned inbound handshake task
* Increase concurrency limit, reduce peer broadcast
* Fix a div_ceil() TODO
* Document security requirements of inbound peer overload handling
* Reduce drop probability and fix its formatting
* Put a 5 second timeout on inbound service requests
* Update drop probability tests
* Add error types, metrics, and logging for InboundTimeout errors
* fix: prevent progress bar from panicking using workaround
* remove more progress bar lengths
* re-add set_len() calls commented out; restore one that shouldn't cause panics
* Update zebra-state/src/service/non_finalized_state.rs
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: teor <teor@riseup.net>
* Implement minor and patch database format versions
* Log and update database format versions when opening database
* Refactor the current list of column families into a constant
* Open all available column families, including from future Zebra versions
* Refactor note commitment tree lookups to go through the height methods
* Make Sapling/Orchard note commitment tree lookup forwards compatible
* Ignore errors reading column family lists from disk
* Update format version comments and TODOs
* Correctly log newly created database formats
* Fix a new cargo lint about resolver versions
* cargo clippy --fix --all-features --all-targets
* cargo fmt --all
* Add missing tokio feature in the state, revealed by the new resolver
* Add missing dev dependencies in zebra-node-services
* Add a missing `tokio` feature from PR #6813
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* updates Cargo.toml
* Migrate to abscissa 0.7.0
* Avoid panic from calling color_eyre::install twice
* Uses 'start' as the default subcommand
* updates default cmd logic
* Fixes minor cli issues
* removes outdated check in acceptance test
* Adds a test for process_cli_args, fixes version_args test.
Adds -V to process_cli_args match case
* Revert "fix(clippy): Silence future-incompat warnings until we upgrade Abscissa (#6024)"
This reverts commit dd90f79b48.
* Drops the worker guard to flush logs when zebra shuts down
* Adds cargo feature to clap
* restores process_cli_args
* updates deny.toml
* Updates EntryPoint help template
* Updates subcommand help msgs
* removes trailing whitespace, capitalizes sentences
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* revert parts of revert "Revert fix(clippy): Silence future-incompat warnings until we upgrade Abscissa"
* Applies suggestions from code review
* Moves EntryPoint to its own module
* fixes version_args test
* Updates changelog
* Prunes redundant test cases
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Revert "Prunes redundant test cases"
This reverts commit 3f73979184.
* Update zebrad/src/commands/entry_point.rs
Co-authored-by: teor <teor@riseup.net>
* Add missing import
* Updates `process_cli_args` to return a result
---------
Co-authored-by: teor <teor@riseup.net>
* Rewrite some state cache docs to clarify
* Add a zebra_network::Config.cache_dir for peer address caches
* Add new config test files and fix config test failure message
* Create some zebra-chain and zebra-network convenience functions
* Add methods for reading and writing the peer address cache
* Add cached disk peers to the initial peers list
* Add metrics and logging for loading and storing the peer cache
* Replace log of useless redacted peer IP addresses
* Limit the peer cache minimum and maximum size, don't write empty caches
* Add a cacheable_peers() method to the address book
* Add a peer disk cache updater task to the peer set tasks
* Document that the peer cache is shared by multiple instances unless configured otherwise
* Disable peer cache read/write in disconnected tests
* Make initial peer cache updater sleep shorter for tests
* Add unit tests for reading and writing the peer cache
* Update the task list in the start command docs
* Modify the existing persistent acceptance test to check for peer caches
* Update the peer cache directory when writing test configs
* Add a CacheDir type so the default config can be enabled, but tests can disable it
* Update tests to use the CacheDir config type
* Rename some CacheDir internals
* Add config file test cases for each kind of CacheDir config
* Panic if the config contains invalid socket addresses, rather than continuing
* Add a network directory to state cache directory contents tests
* Add new network.cache_dir config to the config parsing tests
* rename verifiers
* rename `PreparedBlock` to `SemanticallyVerifiedBlock`
* rename `CommitBlock` to `SemanticallyVerifiedBlock`
* rename `FinalizedBlock` to `CheckpointVerifiedBlock`
* rename `CommitFinalizedBlock` to `CommitCheckpointVerifiedBlock`
* rename `FinalizedWithTrees` to `ContextuallyVerifiedBlockWithTrees`
* rename `ContextuallyValidBlock` to `ContextuallyVerifiedBlock`
* change some `finalized` variables or function arguments to `checkpoint_verified`
* fix docs
* document the difference between `CheckpointVerifiedBlock` and `ContextuallyVerifiedBlock`
* fix doc links
* apply suggestions to request
Co-authored-by: Marek <mail@marek.onl>
* apply suggestions to service
Co-authored-by: Marek <mail@marek.onl>
* apply suggestions to finalized_state.rs and write.rs
Co-authored-by: Marek <mail@marek.onl>
* fmt
* change some more variable names
* change a few missing generics
* fix checkpoint log issue
* rename more `prepared` vars `semantically_verified`
* fix test regex
* fix test regex 2
---------
Co-authored-by: Marek <mail@marek.onl>
* Update MAX_TX_INV_IN_MESSAGE for ZIP-239 WTX IDs
* Combine multiple transaction updates into a single gossip & rate-limit gossips
* Rate-limit block gossips
* Fix mempool_transaction_expiration gossip test timings
* Enforce MAX_TX_INV_IN_MESSAGE in the network layer, rather than each service
* Fix documentation for `Message::Tx`
* Split MAX_INV_IN_RECEIVED_MESSAGE and MAX_TX_INV_IN_SENT_MESSAGE
* Fix log message typo
* Move some docs to/from another PR
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a PeerSocketAddr type which hides its IP address, but shows the port
* Manually replace SocketAddr with PeerSocketAddr where needed
```sh
fastmod SocketAddr PeerSocketAddr zebra-network
```
* Add missing imports
* Make converting into PeerSocketAddr easier
* Fix some unused imports
* Add a canonical_peer_addr() function
* Fix connection handling for PeerSocketAddr
* Fix serialization for PeerSocketAddr
* Fix tests for PeerSocketAddr
* Remove some unused imports
* Fix address book listener handling
* Remove redundant imports and conversions
* Update outdated IPv4-mapped IPv6 address code
* Make addresses canonical when deserializing
* Stop logging peer addresses in RPC code
* Update zebrad tests with new PeerSocketAddr type
* Update zebra-rpc tests with new PeerSocketAddr type
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* add user agent as argument, use git to auto build zebra user agent
* try to fix test
* fix typo
* change expect text
* remove newline
* fix some docs
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* refuse to run Zebra if it is too old
* update the release checklist to consider the constants
* bring newline back
* apply new end of support code
* attempt to add tests (not working yet)
* move eos to progress task
* move tests
* add acceptance test (not working)
* fix tests
* change to block height checks (ugly code)
* change warn days
* refactor estimated blocks per day, etc
* move end of support code to its own task
* change test
* fix some docs
* move constants
* remove uneeded conversions
* downgrade tracing
* reduce end of support time, fix ci changing debugs to info again
* update instructions
* add failure messages
* cargo lock update
* unify releaase name constant
* change info msg
* clippy fixes
* add a block explorer
* ignore testnet in end of support task
* change panic to 16 weeks
* add some documentation about end of support
* Tweak docs wording
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Implement Display and to_string() for NetworkUpgrade
* Add a progress-bar feature to zebrad
* Add the progress bar writer to the tracing component
* Add a block progress bar transmitter
* Correctly shut down the progress bar, and shut it down on an interrupt
* Make it clearer that the progress task never exits
* Add a config for writing logs to a file
* Add a progress-bar feature to zebra-network
* Add a progress bar for the address book size
* Add progress bars for never attempted and failed peers
* Add an optional limit and label to connection counters
* Add open connection progress bars
* Improve CheckpointList API and CheckpointVerifier debugging
* Add checkpoint index and checkpoint queue progress bars
* Security: Limit the number of non-finalized chains tracked by Zebra
* Make some NonFinalizedState methods available with proptest-impl
* Add a non-finalized chain count progress bar
* Track the last fork height for newly forked chains
* Add a should_count_metrics to Chain
* Add a display method for PartialCumulativeWork
* Add a progress bar for each chain fork
* Add a NonFinalizedState::disable_metrics() method and switch to using it
* Move metrics out of Chain because we can't update Arc<Chain>
* Fix: consistently use best chain order when searching chains
* Track Chain progress bars in NonFinalizedState
* Display work as bits, not a multiple of the target difficulty
* Handle negative fork lengths by reporting "No fork"
* Correctly disable unused fork bars
* clippy: rewrite using `match _.cmp(_) { ... }`
* Initial mempool progress bar implementation
* Update Cargo.lock
* Add the actual transaction size as a description to the cost bar
* Only show mempool progress bars after first activation
* Add queued and rejected mempool progress bars
* Clarify cost note is actual size
* Add tracing.log_file config and progress-bar feature to zebrad docs
* Derive Clone for Chain
* Upgrade to howudoin 0.1.2 and remove some bug workarounds
* Directly call the debug formatter to Display a Network
Co-authored-by: Arya <aryasolhi@gmail.com>
* Rename the address count metric to num_addresses
Co-authored-by: Arya <aryasolhi@gmail.com>
* Simplify reverse checkpoint lookup
Co-authored-by: Arya <aryasolhi@gmail.com>
* Simplify progress bar shutdown code
Co-authored-by: Arya <aryasolhi@gmail.com>
* Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE
* Document that the progress task runs forever
* Fix progress log formatting
* If progress-bar is on, log to a file by default
* Create missing directories for log files
* Add file security docs for running Zebra with elevated permissions
* Document automatic log file, spell progress-bar correctly
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* check last seen tip hash from mempool in fetch_mempool_transactions()
* Moves last_seen_tip_hash to ActiveState
* fixes tests and tests fixes
* continues to the next iteration of the loop to make fresh state and mempool requests if called with a long poll id
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: teor <teor@riseup.net>
* adds allow[unused_variable) for linter
* expects a chain tip when not(test)
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Addresses comments in code review
* - Removes second call to `last_tip_change()`
- Fixes tests using enabled mempool
* Adds note about chain tip action requirement to test method `enable()`
* updates doc comment
* Update zebrad/src/components/mempool.rs
Co-authored-by: teor <teor@riseup.net>
* fixes test
---------
Co-authored-by: teor <teor@riseup.net>
* Unify the `impl`s of `Sub` and `Add` for `Height`
* Adjust tests for `Height` subtraction
* Use `Height` instead of `i32`
* Use `block:Height` in RPC tests
* Use `let .. else` statement
Co-authored-by: Arya <aryasolhi@gmail.com>
* Update zebra-consensus/src/block/subsidy/general.rs
* Refactor the handling of height differences
* Remove a redundant comment
* Update zebrad/src/components/sync/progress.rs
Co-authored-by: Arya <aryasolhi@gmail.com>
* Update progress.rs
* impl TryFrom<u32> for Height
* Make some test assertions clearer
* Refactor estimate_up_to()
* Restore a comment that was accidentally removed
* Document when estimate_distance_to_network_chain_tip() returns None
* Change HeightDiff to i64 and make Height.sub(Height) return HeightDiff (no Option)
* Update chain tip estimates for HeightDiff i64
* Update subsidy for HeightDiff i64
* Fix some height calculation test edge cases
* Fix the funding stream interval calculation
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Adds 'Contains' request in state, and:
- adds finalized block hashes to sent_blocks
- replaces Depth call with Contains in sync, inbound, and block verifier
* removes unnecessary From impl
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* updates references to Request::Contains
* Renames zs::response::BlockLocation to KnownBlocks
* Updates AlreadyInChain error
* update docs for sent_hashes.add_finalized
* Update zebra-consensus/src/block.rs
Co-authored-by: teor <teor@riseup.net>
* Update comment for `sent_blocks` field in state service
* update KnownBlock request to check the non-finalized state before responding that a block is in the queue
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* update references to renamed method
* Clear sent_blocks when there's a reset
* Move self.finalized_block_write_sender.is_none() to can_fork_chain_at
* revert changes related to checking queue
---------
Co-authored-by: teor <teor@riseup.net>
* Combine transaction IDs that arrive close together rather than overwriting them
* Reduce the size of transaction ID gossip logs
* Limit the maximum number of times we wait for new changes before sending
* Make logs even shorter
* Expand correctness comment
* Remove trailing space
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* checks tip height before mempool insertions
* adds unit test for reverifying txs
* Adds TODO
* Adds correctness note
* dedup best_tip_height() calls
* Update zebrad/src/components/mempool.rs
Co-authored-by: teor <teor@riseup.net>
* uses Option for expected tip height
* removes misplaced dummy_call()
* calls wait_for_chain_tip without a timeout where it doesn't matter and skips instead of panicking where it doesn't
* Update zebrad/src/components/mempool/tests/vector.rs
* removes whitespace for rustfmt
---------
Co-authored-by: teor <teor@riseup.net>
* Implement the BestChainNextMedianTimePast state request
* Verify the lock times of mempool transactions
* Document that the mempool already handles lock time rejections correctly
* Fix existing tests
* Add new mempool lock time success and failure tests