* print a Zebra logo and some text in progress bar mode
* add network to printed line, add heart to logo
* print logo and message regardless of progress-bar; document how logo was generated
* De-duplicate app_version and user_agent code, rename to build_version
* Make RPC testnet flag forward-compatible with additional testnets
* Fix RPC tests with new argument
* Use "modified" rather than "dirty" for uncommitted changes in build metadata
* Split the vergen version into its own function
* Add a missing timeout to outbound TCP connections
* Move inbound handshakes into their own function, replacing the manual span
* Delete a useless manual span in zebra_network::config
* Add an extra timeout to the spawned inbound handshake task
* Increase concurrency limit, reduce peer broadcast
* Fix a div_ceil() TODO
* Document security requirements of inbound peer overload handling
* Reduce drop probability and fix its formatting
* Put a 5 second timeout on inbound service requests
* Update drop probability tests
* Add error types, metrics, and logging for InboundTimeout errors
* fix: prevent progress bar from panicking using workaround
* remove more progress bar lengths
* re-add set_len() calls commented out; restore one that shouldn't cause panics
* Update zebra-state/src/service/non_finalized_state.rs
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: teor <teor@riseup.net>
* Implement minor and patch database format versions
* Log and update database format versions when opening database
* Refactor the current list of column families into a constant
* Open all available column families, including from future Zebra versions
* Refactor note commitment tree lookups to go through the height methods
* Make Sapling/Orchard note commitment tree lookup forwards compatible
* Ignore errors reading column family lists from disk
* Update format version comments and TODOs
* Correctly log newly created database formats
* Fix a new cargo lint about resolver versions
* cargo clippy --fix --all-features --all-targets
* cargo fmt --all
* Add missing tokio feature in the state, revealed by the new resolver
* Add missing dev dependencies in zebra-node-services
* Add a missing `tokio` feature from PR #6813
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* updates Cargo.toml
* Migrate to abscissa 0.7.0
* Avoid panic from calling color_eyre::install twice
* Uses 'start' as the default subcommand
* updates default cmd logic
* Fixes minor cli issues
* removes outdated check in acceptance test
* Adds a test for process_cli_args, fixes version_args test.
Adds -V to process_cli_args match case
* Revert "fix(clippy): Silence future-incompat warnings until we upgrade Abscissa (#6024)"
This reverts commit dd90f79b48.
* Drops the worker guard to flush logs when zebra shuts down
* Adds cargo feature to clap
* restores process_cli_args
* updates deny.toml
* Updates EntryPoint help template
* Updates subcommand help msgs
* removes trailing whitespace, capitalizes sentences
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* revert parts of revert "Revert fix(clippy): Silence future-incompat warnings until we upgrade Abscissa"
* Applies suggestions from code review
* Moves EntryPoint to its own module
* fixes version_args test
* Updates changelog
* Prunes redundant test cases
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Revert "Prunes redundant test cases"
This reverts commit 3f73979184.
* Update zebrad/src/commands/entry_point.rs
Co-authored-by: teor <teor@riseup.net>
* Add missing import
* Updates `process_cli_args` to return a result
---------
Co-authored-by: teor <teor@riseup.net>
* Rewrite some state cache docs to clarify
* Add a zebra_network::Config.cache_dir for peer address caches
* Add new config test files and fix config test failure message
* Create some zebra-chain and zebra-network convenience functions
* Add methods for reading and writing the peer address cache
* Add cached disk peers to the initial peers list
* Add metrics and logging for loading and storing the peer cache
* Replace log of useless redacted peer IP addresses
* Limit the peer cache minimum and maximum size, don't write empty caches
* Add a cacheable_peers() method to the address book
* Add a peer disk cache updater task to the peer set tasks
* Document that the peer cache is shared by multiple instances unless configured otherwise
* Disable peer cache read/write in disconnected tests
* Make initial peer cache updater sleep shorter for tests
* Add unit tests for reading and writing the peer cache
* Update the task list in the start command docs
* Modify the existing persistent acceptance test to check for peer caches
* Update the peer cache directory when writing test configs
* Add a CacheDir type so the default config can be enabled, but tests can disable it
* Update tests to use the CacheDir config type
* Rename some CacheDir internals
* Add config file test cases for each kind of CacheDir config
* Panic if the config contains invalid socket addresses, rather than continuing
* Add a network directory to state cache directory contents tests
* Add new network.cache_dir config to the config parsing tests
* rename verifiers
* rename `PreparedBlock` to `SemanticallyVerifiedBlock`
* rename `CommitBlock` to `SemanticallyVerifiedBlock`
* rename `FinalizedBlock` to `CheckpointVerifiedBlock`
* rename `CommitFinalizedBlock` to `CommitCheckpointVerifiedBlock`
* rename `FinalizedWithTrees` to `ContextuallyVerifiedBlockWithTrees`
* rename `ContextuallyValidBlock` to `ContextuallyVerifiedBlock`
* change some `finalized` variables or function arguments to `checkpoint_verified`
* fix docs
* document the difference between `CheckpointVerifiedBlock` and `ContextuallyVerifiedBlock`
* fix doc links
* apply suggestions to request
Co-authored-by: Marek <mail@marek.onl>
* apply suggestions to service
Co-authored-by: Marek <mail@marek.onl>
* apply suggestions to finalized_state.rs and write.rs
Co-authored-by: Marek <mail@marek.onl>
* fmt
* change some more variable names
* change a few missing generics
* fix checkpoint log issue
* rename more `prepared` vars `semantically_verified`
* fix test regex
* fix test regex 2
---------
Co-authored-by: Marek <mail@marek.onl>
* Update MAX_TX_INV_IN_MESSAGE for ZIP-239 WTX IDs
* Combine multiple transaction updates into a single gossip & rate-limit gossips
* Rate-limit block gossips
* Fix mempool_transaction_expiration gossip test timings
* Enforce MAX_TX_INV_IN_MESSAGE in the network layer, rather than each service
* Fix documentation for `Message::Tx`
* Split MAX_INV_IN_RECEIVED_MESSAGE and MAX_TX_INV_IN_SENT_MESSAGE
* Fix log message typo
* Move some docs to/from another PR
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a PeerSocketAddr type which hides its IP address, but shows the port
* Manually replace SocketAddr with PeerSocketAddr where needed
```sh
fastmod SocketAddr PeerSocketAddr zebra-network
```
* Add missing imports
* Make converting into PeerSocketAddr easier
* Fix some unused imports
* Add a canonical_peer_addr() function
* Fix connection handling for PeerSocketAddr
* Fix serialization for PeerSocketAddr
* Fix tests for PeerSocketAddr
* Remove some unused imports
* Fix address book listener handling
* Remove redundant imports and conversions
* Update outdated IPv4-mapped IPv6 address code
* Make addresses canonical when deserializing
* Stop logging peer addresses in RPC code
* Update zebrad tests with new PeerSocketAddr type
* Update zebra-rpc tests with new PeerSocketAddr type
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add extra vergen metadata to startup and panic reports
* Upgrade to vergen 8.1.3
* Update build script docs
* Upgrade env var names for vergen 8.1.3
* Switch to `git2` dependency feature
* Use renamed environmental variable for `git describe`
* Ignore vergen errors if `.git` doesn't exist
* Update Cargo.lock
* Remove unused deny.toml exceptions with --all-features
Some of these might be from this PR, or the missed dependency updates.
* add user agent as argument, use git to auto build zebra user agent
* try to fix test
* fix typo
* change expect text
* remove newline
* fix some docs
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* refuse to run Zebra if it is too old
* update the release checklist to consider the constants
* bring newline back
* apply new end of support code
* attempt to add tests (not working yet)
* move eos to progress task
* move tests
* add acceptance test (not working)
* fix tests
* change to block height checks (ugly code)
* change warn days
* refactor estimated blocks per day, etc
* move end of support code to its own task
* change test
* fix some docs
* move constants
* remove uneeded conversions
* downgrade tracing
* reduce end of support time, fix ci changing debugs to info again
* update instructions
* add failure messages
* cargo lock update
* unify releaase name constant
* change info msg
* clippy fixes
* add a block explorer
* ignore testnet in end of support task
* change panic to 16 weeks
* add some documentation about end of support
* Tweak docs wording
---------
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Silence an extremely verbose error in zebra-consensus tests
This disables around 10,000 logs like:
2023-04-18T02:46:28.441662Z WARN init{config=Config { checkpoint_sync: true, debug_skip_parameter_preload: false } network=Mainnet debug_skip_parameter_preload=true}: unexpected error: Closed in state request while verifying previous state checkpoints
* Increase the outbound connection interval to 100ms
* Start the inbound service as soon as possible, and the syncer last
* Increase acceptance test time limits to get more debug info
* Add more debug info to inbound service overload tracing messages
* Implement Display and to_string() for NetworkUpgrade
* Add a progress-bar feature to zebrad
* Add the progress bar writer to the tracing component
* Add a block progress bar transmitter
* Correctly shut down the progress bar, and shut it down on an interrupt
* Make it clearer that the progress task never exits
* Add a config for writing logs to a file
* Add a progress-bar feature to zebra-network
* Add a progress bar for the address book size
* Add progress bars for never attempted and failed peers
* Add an optional limit and label to connection counters
* Add open connection progress bars
* Improve CheckpointList API and CheckpointVerifier debugging
* Add checkpoint index and checkpoint queue progress bars
* Security: Limit the number of non-finalized chains tracked by Zebra
* Make some NonFinalizedState methods available with proptest-impl
* Add a non-finalized chain count progress bar
* Track the last fork height for newly forked chains
* Add a should_count_metrics to Chain
* Add a display method for PartialCumulativeWork
* Add a progress bar for each chain fork
* Add a NonFinalizedState::disable_metrics() method and switch to using it
* Move metrics out of Chain because we can't update Arc<Chain>
* Fix: consistently use best chain order when searching chains
* Track Chain progress bars in NonFinalizedState
* Display work as bits, not a multiple of the target difficulty
* Handle negative fork lengths by reporting "No fork"
* Correctly disable unused fork bars
* clippy: rewrite using `match _.cmp(_) { ... }`
* Initial mempool progress bar implementation
* Update Cargo.lock
* Add the actual transaction size as a description to the cost bar
* Only show mempool progress bars after first activation
* Add queued and rejected mempool progress bars
* Clarify cost note is actual size
* Add tracing.log_file config and progress-bar feature to zebrad docs
* Derive Clone for Chain
* Upgrade to howudoin 0.1.2 and remove some bug workarounds
* Directly call the debug formatter to Display a Network
Co-authored-by: Arya <aryasolhi@gmail.com>
* Rename the address count metric to num_addresses
Co-authored-by: Arya <aryasolhi@gmail.com>
* Simplify reverse checkpoint lookup
Co-authored-by: Arya <aryasolhi@gmail.com>
* Simplify progress bar shutdown code
Co-authored-by: Arya <aryasolhi@gmail.com>
* Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE
* Document that the progress task runs forever
* Fix progress log formatting
* If progress-bar is on, log to a file by default
* Create missing directories for log files
* Add file security docs for running Zebra with elevated permissions
* Document automatic log file, spell progress-bar correctly
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* check last seen tip hash from mempool in fetch_mempool_transactions()
* Moves last_seen_tip_hash to ActiveState
* fixes tests and tests fixes
* continues to the next iteration of the loop to make fresh state and mempool requests if called with a long poll id
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: teor <teor@riseup.net>
* adds allow[unused_variable) for linter
* expects a chain tip when not(test)
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Addresses comments in code review
* - Removes second call to `last_tip_change()`
- Fixes tests using enabled mempool
* Adds note about chain tip action requirement to test method `enable()`
* updates doc comment
* Update zebrad/src/components/mempool.rs
Co-authored-by: teor <teor@riseup.net>
* fixes test
---------
Co-authored-by: teor <teor@riseup.net>
* Unify the `impl`s of `Sub` and `Add` for `Height`
* Adjust tests for `Height` subtraction
* Use `Height` instead of `i32`
* Use `block:Height` in RPC tests
* Use `let .. else` statement
Co-authored-by: Arya <aryasolhi@gmail.com>
* Update zebra-consensus/src/block/subsidy/general.rs
* Refactor the handling of height differences
* Remove a redundant comment
* Update zebrad/src/components/sync/progress.rs
Co-authored-by: Arya <aryasolhi@gmail.com>
* Update progress.rs
* impl TryFrom<u32> for Height
* Make some test assertions clearer
* Refactor estimate_up_to()
* Restore a comment that was accidentally removed
* Document when estimate_distance_to_network_chain_tip() returns None
* Change HeightDiff to i64 and make Height.sub(Height) return HeightDiff (no Option)
* Update chain tip estimates for HeightDiff i64
* Update subsidy for HeightDiff i64
* Fix some height calculation test edge cases
* Fix the funding stream interval calculation
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Adds 'Contains' request in state, and:
- adds finalized block hashes to sent_blocks
- replaces Depth call with Contains in sync, inbound, and block verifier
* removes unnecessary From impl
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* updates references to Request::Contains
* Renames zs::response::BlockLocation to KnownBlocks
* Updates AlreadyInChain error
* update docs for sent_hashes.add_finalized
* Update zebra-consensus/src/block.rs
Co-authored-by: teor <teor@riseup.net>
* Update comment for `sent_blocks` field in state service
* update KnownBlock request to check the non-finalized state before responding that a block is in the queue
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* update references to renamed method
* Clear sent_blocks when there's a reset
* Move self.finalized_block_write_sender.is_none() to can_fork_chain_at
* revert changes related to checking queue
---------
Co-authored-by: teor <teor@riseup.net>
* Combine transaction IDs that arrive close together rather than overwriting them
* Reduce the size of transaction ID gossip logs
* Limit the maximum number of times we wait for new changes before sending
* Make logs even shorter
* Expand correctness comment
* Remove trailing space
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Allow missing docs directly on derived error types
* Make Request::BestChainBlockHash redirect to the ReadStateService
* Re-write the checkpoint_sync documentation based on the latest consensus rules
* Expose the underlying iterator for CheckpointList
* Validate existing state block hashes at startup, but ignore the result
* Monitor state block hash checkpoint task in the start command
* Fix indentation
* Make logging consistent
* Explain the config needed for full security
* Tidy required checkpoints docs, expand other docs
* Add security and deprecation changelog entries
* Replace task handle vector with a struct
* Add a test that this consensus-critical code actually runs and finishes
* Make some state methods and types available in tests
* Add missing docs
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* checks tip height before mempool insertions
* adds unit test for reverifying txs
* Adds TODO
* Adds correctness note
* dedup best_tip_height() calls
* Update zebrad/src/components/mempool.rs
Co-authored-by: teor <teor@riseup.net>
* uses Option for expected tip height
* removes misplaced dummy_call()
* calls wait_for_chain_tip without a timeout where it doesn't matter and skips instead of panicking where it doesn't
* Update zebrad/src/components/mempool/tests/vector.rs
* removes whitespace for rustfmt
---------
Co-authored-by: teor <teor@riseup.net>
* Implement the BestChainNextMedianTimePast state request
* Verify the lock times of mempool transactions
* Document that the mempool already handles lock time rejections correctly
* Fix existing tests
* Add new mempool lock time success and failure tests
* Derive default using #[default]
* Implement PartialEq manually to satisfy clippy
* Allow a manual derive in test-only code
* Fix some missing docs warnings in the Docker build
* adds ValidateBlock request to state
* adds `Request` enum in block verifier
skips solution check for BlockProposal requests
calls CheckBlockValidity instead of Commit block for BlockProposal requests
* uses new Request in references to chain verifier
* adds getblocktemplate proposal mode response type
* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state
* Adds PR review revisions
* adds info log in CheckBlockProposalValidity
* Reverts replacement of match statement
* adds `GetBlockTemplate::capabilities` fn
* conditions calling checkpoint verifier on !request.is_proposal
* updates references to validate_and_commit_non_finalized
* adds snapshot test, updates test vectors
* adds `should_count_metrics` to NonFinalizedState
* Returns an error from chain verifier for block proposal requests below checkpoint height
adds feature flags
* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD
* adds back block::Request to zebra-consensus lib
* updates snapshots
* Removes unnecessary network arg
* skips req in tracing intstrument for read state
* Moves out block proposal validation to its own fn
* corrects `difficulty_threshold_is_valid` docs
adds/fixes some comments, adds TODOs
general cleanup from a self-review.
* Update zebra-state/src/service.rs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: teor <teor@riseup.net>
* check best chain tip
* Update zebra-state/src/service.rs
Co-authored-by: teor <teor@riseup.net>
* Applies cleanup suggestions from code review
* updates gbt acceptance test to make a block proposal
* fixes json parsing mistake
* adds retries
* returns reject reason if there are no retries left
* moves result deserialization to RPCRequestClient method, adds docs, moves jsonrpc_core to dev-dependencies
* moves sleep(EXPECTED_TX_TIME) out of loop
* updates/adds info logs in retry loop
* Revert "moves sleep(EXPECTED_TX_TIME) out of loop"
This reverts commit f7f0926f4050519687a79afc16656c3f345c004b.
* adds `allow(dead_code)`
* tests with curtime, mintime, & maxtime
* Fixes doc comment
* Logs error responses from chain_verifier CheckProposal requests
* Removes retry loop, adds num_txs log
* removes verbose info log
* sorts mempool_txs before generating merkle root
* Make imports conditional on a feature
* Disable new CI tests until bugs are fixed
* adds support for getpeerinfo RPC
* adds vector test
* Always passes address_book to RpcServer
* Update zebra-network/src/address_book.rs
Co-authored-by: teor <teor@riseup.net>
* Renames PeerObserver to AddressBookPeers
* adds getpeerinfo acceptance test
* Asserts that addresses parse into SocketAddr
* adds launches_lightwalletd field to TestType::LaunchWithEmptyState and uses it from the get_peer_info test
* renames peer_observer mod
* uses SocketAddr as `addr` field type in PeerInfo
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* adds ValidateBlock request to state
* adds `Request` enum in block verifier
skips solution check for BlockProposal requests
calls CheckBlockValidity instead of Commit block for BlockProposal requests
* uses new Request in references to chain verifier
* adds getblocktemplate proposal mode response type
* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state
* Adds PR review revisions
* adds info log in CheckBlockProposalValidity
* Reverts replacement of match statement
* adds `GetBlockTemplate::capabilities` fn
* conditions calling checkpoint verifier on !request.is_proposal
* updates references to validate_and_commit_non_finalized
* adds snapshot test, updates test vectors
* adds `should_count_metrics` to NonFinalizedState
* Returns an error from chain verifier for block proposal requests below checkpoint height
adds feature flags
* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD
* adds back block::Request to zebra-consensus lib
* updates snapshots
* Removes unnecessary network arg
* skips req in tracing intstrument for read state
* Moves out block proposal validation to its own fn
* corrects `difficulty_threshold_is_valid` docs
adds/fixes some comments, adds TODOs
general cleanup from a self-review.
* Update zebra-state/src/service.rs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: teor <teor@riseup.net>
* check best chain tip
* Update zebra-state/src/service.rs
Co-authored-by: teor <teor@riseup.net>
* Applies cleanup suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add some TODOs
* Move and rename height_from_signed_int()
* Move get_block_template() support functions to a submodule
* Fix incorrect P2SH comments and logs
* Split initial checks into their own functions
* Split state fetch into its own function, do some cleanup
* Move get_block_template_opts to get_block_template::parameters
* Fix and simplify test imports
* Rename block_height to next_block_height
* Rename to chain_tip_and_local_time to make it clear what it contains
* Split fetching mempool transactions out, include them in long poll id
* Refactor coinbase generation
* Split default root calculation into a separate function
* Use DateTime32 for getblocktemplate times
* Use typed difficulty fields rather than strings
* Split out a generate coinbase and roots function
* Move GetBlockTemplate construction into a method
* Document what happens to unusual difficulty values
* Clean up some TODOs
* fastmod check_address check_miner_address
* cargo fmt --all
* Fix an incorrect panic message
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move Drop from mempool::ActiveState to mempool::Downloads, to avoid bugs
* Re-verify mempool transactions after a fork
And add a marker struct for mempool download cancellation.
* Update README based on recent mitigations for some issues, tidy format
* Make mempool proptests easier to debug
* Make UnminedTx Display text much smaller
* Update tests for mempool transaction re-verification after forks
* Retry all stored and pending transactions
* Fix a test to check for mempool reset retries
* Remove verbose continuous_blockchain test logs
* Downgrade verbose zebra-network logs to debug
* Downgrade some state logs to debug during tests
* Mark were we would add always-on log filters, if we needed to
* Reduce the number of mempool property tests, to reduce logging
teor
Conrado Gouvea
Arya
Deirdre Connolly
Alfredo Garcia
Marek