* Add extra test type modes to support zebra-checkpoints
* Add Mainnet and Testnet zebra-checkpoints test harnesses
* Add zebra-checkpoints to test docker images
* Add zebra-checkpoints test entrypoints
* Add Mainnet CI workflow for zebra-checkpoints
* Enable zebra-checkpoints feature in the test image
* Use the same features for (almost) all the docker tests
* Make workflow features match Docker features
* Add a feature note
* Add a zebra-checkpoints test feature to zebrad
* Remove the "no cached state" testnet code
* Log a startup message to standard error when launching zebra-checkpoints
* Rename tests to avoid partial name conflicts
* Fix log formatting
* Add sentry feature to experimental docker image build
* Explain what ENTRYPOINT_FEATURES is used for
* Use the correct zebra-checkpoints path
* Silence zebrad logs while generating checkpoints
* Fix zebra-checkpoints log handling
* Re-enable waiting for zebrad to fully sync
* Add documentation for how to run these tests individually
* Start generating checkpoints from the last compiled-in checkpoint
* Fix clippy lints
* Revert changes to TestType
* Wait for all the checkpoints before finishing
* Add more stderr debugging to zebra-checkpoints
* Fix an outdated module comment
* Add a workaround for zebra-checkpoints launch/run issues
* Use temp dir and log what it is
* Log extra metadata about the zebra-checkpoints binary
* Add note about unstable feature -Z bindeps
* Temporarily make the test run faster and with debug info
* Log the original test command name when showing stdout and stderr
* Try zebra-checkpoints in the system path first, then the cargo path
* Fix slow thread close bug in dual process test harness
* If the logs are shown, don't say they are hidden
* Run `zebra-checkpoints --help` to work out what's going on in CI
* Build `zebra-utils` binaries for `zebrad` integration tests
* Revert temporary debugging changes
* Revert changes that were moved to another PR
* Rename variables so it's clearer what they do
* Fully document zebra-checkpoints arguments
* Remove some outdated references to zcashd
* Add a json-conversion feature for converting JSON to valid Heights
* Simplify zebra-checkpoints code using conversion methods
* Track the last checkpoint height rather than the height gap
* Move all the CLI-specific code into a single function
* Remove cfg(feature) from the test RPC client API
* Move the RpcRequestClient into zebra-node-services
* Fix the case of RpcRequestClient
* Add transport and addr arguments to zebra-checkpoints
* Make zebra-checkpoints support both CLI and direct JSON-RPC connections
* Fix RpcRequestClient compilation
* Add a suggestion for zcashd authentication failures
* Document required features
* Handle differences in CLI & direct parameter and response formats
* Replace a custom function with an existing dependency function
* Add a checkpoint list test for minimum height gaps
* Silence an extremely verbose error in zebra-consensus tests
This disables around 10,000 logs like:
2023-04-18T02:46:28.441662Z WARN init{config=Config { checkpoint_sync: true, debug_skip_parameter_preload: false } network=Mainnet debug_skip_parameter_preload=true}: unexpected error: Closed in state request while verifying previous state checkpoints
* Increase the outbound connection interval to 100ms
* Start the inbound service as soon as possible, and the syncer last
* Increase acceptance test time limits to get more debug info
* Add more debug info to inbound service overload tracing messages
* Implement Display and to_string() for NetworkUpgrade
* Add a progress-bar feature to zebrad
* Add the progress bar writer to the tracing component
* Add a block progress bar transmitter
* Correctly shut down the progress bar, and shut it down on an interrupt
* Make it clearer that the progress task never exits
* Add a config for writing logs to a file
* Add a progress-bar feature to zebra-network
* Add a progress bar for the address book size
* Add progress bars for never attempted and failed peers
* Add an optional limit and label to connection counters
* Add open connection progress bars
* Improve CheckpointList API and CheckpointVerifier debugging
* Add checkpoint index and checkpoint queue progress bars
* Security: Limit the number of non-finalized chains tracked by Zebra
* Make some NonFinalizedState methods available with proptest-impl
* Add a non-finalized chain count progress bar
* Track the last fork height for newly forked chains
* Add a should_count_metrics to Chain
* Add a display method for PartialCumulativeWork
* Add a progress bar for each chain fork
* Add a NonFinalizedState::disable_metrics() method and switch to using it
* Move metrics out of Chain because we can't update Arc<Chain>
* Fix: consistently use best chain order when searching chains
* Track Chain progress bars in NonFinalizedState
* Display work as bits, not a multiple of the target difficulty
* Handle negative fork lengths by reporting "No fork"
* Correctly disable unused fork bars
* clippy: rewrite using `match _.cmp(_) { ... }`
* Initial mempool progress bar implementation
* Update Cargo.lock
* Add the actual transaction size as a description to the cost bar
* Only show mempool progress bars after first activation
* Add queued and rejected mempool progress bars
* Clarify cost note is actual size
* Add tracing.log_file config and progress-bar feature to zebrad docs
* Derive Clone for Chain
* Upgrade to howudoin 0.1.2 and remove some bug workarounds
* Directly call the debug formatter to Display a Network
Co-authored-by: Arya <aryasolhi@gmail.com>
* Rename the address count metric to num_addresses
Co-authored-by: Arya <aryasolhi@gmail.com>
* Simplify reverse checkpoint lookup
Co-authored-by: Arya <aryasolhi@gmail.com>
* Simplify progress bar shutdown code
Co-authored-by: Arya <aryasolhi@gmail.com>
* Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE
* Document that the progress task runs forever
* Fix progress log formatting
* If progress-bar is on, log to a file by default
* Create missing directories for log files
* Add file security docs for running Zebra with elevated permissions
* Document automatic log file, spell progress-bar correctly
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* check last seen tip hash from mempool in fetch_mempool_transactions()
* Moves last_seen_tip_hash to ActiveState
* fixes tests and tests fixes
* continues to the next iteration of the loop to make fresh state and mempool requests if called with a long poll id
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: teor <teor@riseup.net>
* adds allow[unused_variable) for linter
* expects a chain tip when not(test)
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Addresses comments in code review
* - Removes second call to `last_tip_change()`
- Fixes tests using enabled mempool
* Adds note about chain tip action requirement to test method `enable()`
* updates doc comment
* Update zebrad/src/components/mempool.rs
Co-authored-by: teor <teor@riseup.net>
* fixes test
---------
Co-authored-by: teor <teor@riseup.net>
* Unify the `impl`s of `Sub` and `Add` for `Height`
* Adjust tests for `Height` subtraction
* Use `Height` instead of `i32`
* Use `block:Height` in RPC tests
* Use `let .. else` statement
Co-authored-by: Arya <aryasolhi@gmail.com>
* Update zebra-consensus/src/block/subsidy/general.rs
* Refactor the handling of height differences
* Remove a redundant comment
* Update zebrad/src/components/sync/progress.rs
Co-authored-by: Arya <aryasolhi@gmail.com>
* Update progress.rs
* impl TryFrom<u32> for Height
* Make some test assertions clearer
* Refactor estimate_up_to()
* Restore a comment that was accidentally removed
* Document when estimate_distance_to_network_chain_tip() returns None
* Change HeightDiff to i64 and make Height.sub(Height) return HeightDiff (no Option)
* Update chain tip estimates for HeightDiff i64
* Update subsidy for HeightDiff i64
* Fix some height calculation test edge cases
* Fix the funding stream interval calculation
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Adds 'Contains' request in state, and:
- adds finalized block hashes to sent_blocks
- replaces Depth call with Contains in sync, inbound, and block verifier
* removes unnecessary From impl
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* updates references to Request::Contains
* Renames zs::response::BlockLocation to KnownBlocks
* Updates AlreadyInChain error
* update docs for sent_hashes.add_finalized
* Update zebra-consensus/src/block.rs
Co-authored-by: teor <teor@riseup.net>
* Update comment for `sent_blocks` field in state service
* update KnownBlock request to check the non-finalized state before responding that a block is in the queue
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* update references to renamed method
* Clear sent_blocks when there's a reset
* Move self.finalized_block_write_sender.is_none() to can_fork_chain_at
* revert changes related to checking queue
---------
Co-authored-by: teor <teor@riseup.net>
* build(deps): bump dirs from 4.0.0 to 5.0.0
Bumps [dirs](https://github.com/soc/dirs-rs) from 4.0.0 to 5.0.0.
- [Release notes](https://github.com/soc/dirs-rs/releases)
- [Commits](https://github.com/soc/dirs-rs/commits)
---
updated-dependencies:
- dependency-name: dirs
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* Allow a different dirs-sys version used for a different purpose
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* add initial ES support
* hide elasticsearch behind feture, add config
* make the builds and clippy happy
* move ES code to a function
* change database name
* fix database name and panic
* increase close to tip interval
* update deny for elastic
* remove a block clone
* try to fix builds
* fix builds 2
* refactor some imports and unwraps
* typo
* change argument of elasticsearch function
* move elastic call to the end of `commit_finalized_direct`
* get height from block
* Combine transaction IDs that arrive close together rather than overwriting them
* Reduce the size of transaction ID gossip logs
* Limit the maximum number of times we wait for new changes before sending
* Make logs even shorter
* Expand correctness comment
* Remove trailing space
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Remove unused dependencies
* Check for newly unused dependencies in CI
* Use the correct grep command
* Always show cargo machete output
* Ignore cargo machete exit status, use grep instead
* Use if instead of && and subshells
* Invert if logic
* Mark Zebra coinbase transactions with extra coinbase data
* Log when we commit a block mined by Zebra to our state
* Reduce logging instrumentation during block writes
* Remove debug types in Zebra block log
* Add network and commit to write task logs
* Apply an allow-list before we log arbitrary user messages from blocks
* Rate-limit Zebra mined block logging to once every 1000 blocks
* Add mining configs for extra coinbase data and imitating zcashd, but don't use them yet
* Check CoinbaseData size limit when building transparent transactions
* Replace LIKE_ZCASHD constants with a config
* Take extra coinbase data from the configured string
* Update the zebrad configs in the tests with new config fields
* Allow missing docs directly on derived error types
* Make Request::BestChainBlockHash redirect to the ReadStateService
* Re-write the checkpoint_sync documentation based on the latest consensus rules
* Expose the underlying iterator for CheckpointList
* Validate existing state block hashes at startup, but ignore the result
* Monitor state block hash checkpoint task in the start command
* Fix indentation
* Make logging consistent
* Explain the config needed for full security
* Tidy required checkpoints docs, expand other docs
* Add security and deprecation changelog entries
* Replace task handle vector with a struct
* Add a test that this consensus-critical code actually runs and finishes
* Make some state methods and types available in tests
* Add missing docs
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* checks tip height before mempool insertions
* adds unit test for reverifying txs
* Adds TODO
* Adds correctness note
* dedup best_tip_height() calls
* Update zebrad/src/components/mempool.rs
Co-authored-by: teor <teor@riseup.net>
* uses Option for expected tip height
* removes misplaced dummy_call()
* calls wait_for_chain_tip without a timeout where it doesn't matter and skips instead of panicking where it doesn't
* Update zebrad/src/components/mempool/tests/vector.rs
* removes whitespace for rustfmt
---------
Co-authored-by: teor <teor@riseup.net>
* Stabilise the BestChainBlockHash state request
* Always include the block hash in the getblock RPC response
* Make the lightwalletd integration tests compatible with zcash/lightwalletd
* Update getblock RPC snapshots
* Return the correct missing block error code
* adds basic usage of long polling in gbt test
* adds !submit_old check before cancelling proposing a new template
* Removes break statement in long polling task
* Update zebrad/tests/common/rpc_client.rs
Co-authored-by: teor <teor@riseup.net>
* use blocking_send and watch channel
* fix "cannot block the current thread from within a runtime"
* Reduces interval between proposals and increases num proposals required.
* Runs rate-limiting sleeps in parallel to validation
* corrects comment.
---------
Co-authored-by: teor <teor@riseup.net>
* Revert "Temporarily fix the code for a disabled test"
This reverts commit d915624417cc946e53aac76449e0b5b719e03d2a.
* Check every valid time source in the proposal tests
* Activate block proposal tests in CI
* Repeats block proposal test a few times at an interval to sample different mempool contents
* Increase the number of templates tested to 10
Co-authored-by: Arya <aryasolhi@gmail.com>
---------
Co-authored-by: arya2 <aryasolhi@gmail.com>
* bump v1.0.0-rc.4
* Add new mainnet and testnet checkpoints generated using zcashd
* make changes to changelog from review
* move item to new section
* change section name
---------
Co-authored-by: teor <teor@riseup.net>
* Implement the BestChainNextMedianTimePast state request
* Verify the lock times of mempool transactions
* Document that the mempool already handles lock time rejections correctly
* Fix existing tests
* Add new mempool lock time success and failure tests
* Revert "Update code that we're going to delete in the next PR anyway"
This reverts commit 1fed70da9e.
* Initial zcash-test-block-template script without block proposal construction
* Try creating block data using jq and printf
* Move proposal_block_from_template() to zebra-rpc and remove eyre dependency
* Implement FromStr for DateTime32 and Duration32
* Basic block-template-to-proposal command without time source handling
* Move block proposal code into its own module
* Use time source in block-template-to-proposal
* Make block-template-to-proposal require the getblocktemplate-rpcs feature
* Use block-template-to-proposal in the test script zcash-rpc-block-template-to-proposal
* Apply new hex formatting to commitments and nonces in block proposal tests
* Re-add missing imports from rebase
* Re-add missing conversions from rebase
* Update to new method name after rebase
* Derive a Default impl
* Restore a comment that was accidentally deleted during the rebase
* Avoid a clippy::unwrap-in-result
* Temporarily fix the code for a disabled test
* Fix tool build with Docker caches
* Make clippy happy with the temporary fix
* Give a pass/fail status for each proposal response
* Accept zcashd block templates in block-template-to-proposal
* Fix pretty printing
* Zebra expects a longpollid, so give it a dummy value
* Add "required" fields which Zebra requires
* Use curtime as the dummy maxtime in zcashd templates
* Support large block proposals by reading proposal data from a file
* Test all valid time modes for each proposal
* Allow the user to set the time command
* Put debug logs into their own files
* Exit with an error status when any proposal is invalid
* Log the date and time to make it easier to match errors to node logs
* Derive default using #[default]
* Implement PartialEq manually to satisfy clippy
* Allow a manual derive in test-only code
* Fix some missing docs warnings in the Docker build
* Format commitments and nonces as hex
* Format Sprout and Sapling nullifiers as hex when debugging
* Format Sprout commitments as hex when debugging
* Format redpallas keys as hex when debugging
* Update code that we're going to delete in the next PR anyway
* adds ValidateBlock request to state
* adds `Request` enum in block verifier
skips solution check for BlockProposal requests
calls CheckBlockValidity instead of Commit block for BlockProposal requests
* uses new Request in references to chain verifier
* adds getblocktemplate proposal mode response type
* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state
* Adds PR review revisions
* adds info log in CheckBlockProposalValidity
* Reverts replacement of match statement
* adds `GetBlockTemplate::capabilities` fn
* conditions calling checkpoint verifier on !request.is_proposal
* updates references to validate_and_commit_non_finalized
* adds snapshot test, updates test vectors
* adds `should_count_metrics` to NonFinalizedState
* Returns an error from chain verifier for block proposal requests below checkpoint height
adds feature flags
* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD
* adds back block::Request to zebra-consensus lib
* updates snapshots
* Removes unnecessary network arg
* skips req in tracing intstrument for read state
* Moves out block proposal validation to its own fn
* corrects `difficulty_threshold_is_valid` docs
adds/fixes some comments, adds TODOs
general cleanup from a self-review.
* Update zebra-state/src/service.rs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: teor <teor@riseup.net>
* check best chain tip
* Update zebra-state/src/service.rs
Co-authored-by: teor <teor@riseup.net>
* Applies cleanup suggestions from code review
* updates gbt acceptance test to make a block proposal
* fixes json parsing mistake
* adds retries
* returns reject reason if there are no retries left
* moves result deserialization to RPCRequestClient method, adds docs, moves jsonrpc_core to dev-dependencies
* moves sleep(EXPECTED_TX_TIME) out of loop
* updates/adds info logs in retry loop
* Revert "moves sleep(EXPECTED_TX_TIME) out of loop"
This reverts commit f7f0926f4050519687a79afc16656c3f345c004b.
* adds `allow(dead_code)`
* tests with curtime, mintime, & maxtime
* Fixes doc comment
* Logs error responses from chain_verifier CheckProposal requests
* Removes retry loop, adds num_txs log
* removes verbose info log
* sorts mempool_txs before generating merkle root
* Make imports conditional on a feature
* Disable new CI tests until bugs are fixed
* adds support for getpeerinfo RPC
* adds vector test
* Always passes address_book to RpcServer
* Update zebra-network/src/address_book.rs
Co-authored-by: teor <teor@riseup.net>
* Renames PeerObserver to AddressBookPeers
* adds getpeerinfo acceptance test
* Asserts that addresses parse into SocketAddr
* adds launches_lightwalletd field to TestType::LaunchWithEmptyState and uses it from the get_peer_info test
* renames peer_observer mod
* uses SocketAddr as `addr` field type in PeerInfo
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* adds ValidateBlock request to state
* adds `Request` enum in block verifier
skips solution check for BlockProposal requests
calls CheckBlockValidity instead of Commit block for BlockProposal requests
* uses new Request in references to chain verifier
* adds getblocktemplate proposal mode response type
* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state
* Adds PR review revisions
* adds info log in CheckBlockProposalValidity
* Reverts replacement of match statement
* adds `GetBlockTemplate::capabilities` fn
* conditions calling checkpoint verifier on !request.is_proposal
* updates references to validate_and_commit_non_finalized
* adds snapshot test, updates test vectors
* adds `should_count_metrics` to NonFinalizedState
* Returns an error from chain verifier for block proposal requests below checkpoint height
adds feature flags
* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD
* adds back block::Request to zebra-consensus lib
* updates snapshots
* Removes unnecessary network arg
* skips req in tracing intstrument for read state
* Moves out block proposal validation to its own fn
* corrects `difficulty_threshold_is_valid` docs
adds/fixes some comments, adds TODOs
general cleanup from a self-review.
* Update zebra-state/src/service.rs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: teor <teor@riseup.net>
* check best chain tip
* Update zebra-state/src/service.rs
Co-authored-by: teor <teor@riseup.net>
* Applies cleanup suggestions from code review
* updates gbt acceptance test to make a block proposal
* fixes json parsing mistake
* adds retries
* returns reject reason if there are no retries left
* moves result deserialization to RPCRequestClient method, adds docs, moves jsonrpc_core to dev-dependencies
* moves sleep(EXPECTED_TX_TIME) out of loop
* updates/adds info logs in retry loop
* Revert "moves sleep(EXPECTED_TX_TIME) out of loop"
This reverts commit f7f0926f4050519687a79afc16656c3f345c004b.
* adds `allow(dead_code)`
* tests with curtime, mintime, & maxtime
* Fixes doc comment
* Logs error responses from chain_verifier CheckProposal requests
* Removes retry loop, adds num_txs log
* removes verbose info log
* sorts mempool_txs before generating merkle root
* Make imports conditional on a feature
* Disable new CI tests until bugs are fixed
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* adds ValidateBlock request to state
* adds `Request` enum in block verifier
skips solution check for BlockProposal requests
calls CheckBlockValidity instead of Commit block for BlockProposal requests
* uses new Request in references to chain verifier
* adds getblocktemplate proposal mode response type
* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state
* Adds PR review revisions
* adds info log in CheckBlockProposalValidity
* Reverts replacement of match statement
* adds `GetBlockTemplate::capabilities` fn
* conditions calling checkpoint verifier on !request.is_proposal
* updates references to validate_and_commit_non_finalized
* adds snapshot test, updates test vectors
* adds `should_count_metrics` to NonFinalizedState
* Returns an error from chain verifier for block proposal requests below checkpoint height
adds feature flags
* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD
* adds back block::Request to zebra-consensus lib
* updates snapshots
* Removes unnecessary network arg
* skips req in tracing intstrument for read state
* Moves out block proposal validation to its own fn
* corrects `difficulty_threshold_is_valid` docs
adds/fixes some comments, adds TODOs
general cleanup from a self-review.
* Update zebra-state/src/service.rs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update zebra-rpc/src/methods/get_block_template_rpcs.rs
Co-authored-by: teor <teor@riseup.net>
* check best chain tip
* Update zebra-state/src/service.rs
Co-authored-by: teor <teor@riseup.net>
* Applies cleanup suggestions from code review
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Changelog with trivial entries
* Delete trivial entries
* Summarise known issues in README, but don't change the list yet
* Add block timeouts to known issues
* Update the release template to add missing version files
* Bump crate versions
* Add the required Rust version to the release checklist
* Update the Rust version requirement to 1.65, Zebra now uses `let ... else ...`
* Update checkpoints
* Add checkpoints to the CHANGELOG
* Breaking Rust compiler version change
* Clarify the latest stable supported rust version