* Split out ReadRequest and ReadResponse state service enums
* Simplify RPC test vectors
* Split state requests into Request and ReadRequest
* Make zebra-rpc use the new state ReadRequest
* Add request metrics to ReadStateService
Add a field to distinguish StateService and ReadStateService metrics.
Add missing StateService request metrics.
* Refactor state transaction lookup so it can be shared between services
* Implement ReadState Request::Transaction
* Revert "Drop the Chain watch channel lock before accessing the finalized state"
This reverts commit 8870944d13.
* Add a WatchReceiver wrapper that always clones the borrowed watch data
This avoids deadlocks, by holding the read lock for as short a time as possible.
* Drop the shared Arc<Chain>s as quickly as possible
This reduces memory usage.
* Make read::block more flexible, by accepting any AsRef<Chain>
* Make the block method docs consistent
* Avoid livelocks by explicitly dropping the borrow after the clone
* Use the read-only state service for RPCs
* Refactor non-finalized block lookup into Chain
* Implement the read-only state block request
* Drop the Chain watch channel lock before accessing the finalized state
* Query Sapling & Orchard trees by height in the finalized state
* Add Sapling & Orchard trees to the non-finalized state
* Add a TODO about concurrent read-only access to Sprout tree
Co-authored-by: teor <teor@riseup.net>
* Update the database format version
* Keep only the most recent Sprout tree in the database
* Check that the database returns empty trees for the genesis block
* Assert that the database returns the highest trees
* Document how to update insta snapshots
* Add note commitment tree insta snapshot tests
* Add comments about cached tree roots in snapshots
* Add snapshot data for sapling and orchard trees
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* Add state service module docs and cleanup
* Move and add finalized state methods
* Add chain and non-finalized state methods
* Cleanup methods and imports
* Create a ReadStateService type
* Add a stub service implementation
* Add a TODO
* Update ReadStateService request stubs with RPC names and tickets
* Documentation updates
* Make RPC State generic bounds accept a buffered state and a read-only state
* Doc updates
* Add missing proptest-impl feature in RPC dev dependencies
* Move the legacy chain check to the `check` module
And move `populated_state` to the `arbitrary` module.
* Cleanup imports
* Document the state service struct
* Split state block iter into its own module
* Prepare the finalized state for read-only state
* Add a forced shutdown mode, used in test code before forced exits
* Document the small database drop race condition window
* Move the legacy chain check to the `check` module
And move `populated_state` to the `arbitrary` module.
* Cleanup imports
* Document the state service struct
* Split state block iter into its own module
* Drop chain if it becomes invalid
Avoid returning a chain that shouldn't be used again by having the
method take ownership of `self` and dropping it on error.
* Update documentation and comments
Restore them to be closer to how they were on `main`.
* fix(shielded): use RwLock for note commitment tree root caches
* Prepare the non-finalized state for read-only state
* fix(state): performance tweak for chain forks
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feature(rpc): start adding a `getblock` method
* fix(rpc): replace oneshot
* fix(rpc): replace a panic with error
* fix(rpc): fix test
* feature(rpc): add hex to response
* refactor(rpc): use generic instead of alias
* docs(rpc): improve docs for getblock method
* test(rpc): add a test for getblock method
* deps(rpc): remove non needed tower features
Co-authored-by: teor <teor@riseup.net>
* docs(rpc): add a note to getblock doc
* refactor(rpc): replace alias
* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format
* tests(rpc): add network argument to `populated_state()`
* refactor(rpc): use an error for state service readiness
* fix(rpc): add parameter
* fix(rpc): clippy
* nit(rpc): remove new line from imports
* fix(rpc): remove commented code
* fix(rpc): simplify error
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use a `SerializedBlock` type to help serializing blocks (#3725)
* Create a `SerializedBlock` helper type
Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.
* Use `into_iter` instead of `iter`
There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.
* Use `SerializedBlock` in `GetBlock`
Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.
The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.
* feature(rpc): add suggestions from code review
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* tests(rpc): make sure mempool has no requests in get_block test
* fix(rpc): change height argument type in getblock method
* fix(rpc): rustfmt
* fix(rpc): replace panic
* fix(rpc): change getblock response
* fix(rpc): fix lightwalletd test
* tests(rpc): add a getblock error test
* fix(rpc): try another regex
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* refactor(test/block): rename large single transaction function
```sh
fastmod single_transaction_block single_transaction_block_many_inputs
```
* rustfmt
* test(block): add a test block with many transparent outputs
* doc(db): explain why we can't just get the UTXOs right before they are deleted
* refactor(db): split out a block data write method
* refactor(block): add a height argument to new_outputs
* test(db): add block and transaction round-trip tests
Including large blocks and transactions.
* test(db): fix large block serialization instability in the tests
* doc(block): add TODOs for generating correct blocks
* Make transparent output functions which take a height test-only
* make sure generated blocks are actually over/under-sized
* replace println!() with an error!() log
* refactor(db): simplify block height serialization
* refactor(db): make height serialization length generic
* refactor(db): create a TransactionIndex type
This changes the names of some snapshot types,
but doesn't change any data.
* refactor(db): create transparent OutputIndex and OutputLocation types
This keeps the same serialization, to avoid changing the database version.
* doc(rfc/db): make transparent database type names consistent
* doc(rfc/db): fix a bug in the Utxo.is_coinbase derivation
* fix(db): use the correct serialized size for OutputLocation
* doc(db): fix some comments
* refactor(db): split disk serialization types into their own module
* refactor(db): split the disk format into modules
* doc(db/test): explain the RON serialization format
* Upgrade some dependencies
* Upgrade some dependencies
* Upgrade dependencies for zebrad
* Upgrade tracing dependencies
* Revert `tor` & `arti`
* Upgrade `criterion` & `pin-project` in `deny.toml`
* Remove some dependencies from `skip-tree` in `deny.toml`
* Revert some the versions of dependencies because of duplicates
* Revert proptest regressions
* Upgrade dependencies, then ignore some more duplicates (#3716)
* feat(actions)!: add full sync test (#3582)
* add(tests): full sync test
* fix(test): add build
* fix(deploy): escape double dashes '--' correctly
* fix(test): remove unexpected --no-capture arg
error: Found argument '--nocapture' which wasn't expected, or isn't valid in this context
* refactor(docker): use default executable as entrypoint
* refactor(startup): add a custom entrypoint
* fix(test): add missing TEST_FULL_SYNC variable
* test(timeout): use the biggest machine
* fix
* fix(deploy): use latest successful image
* typo
* refactor(docker): generate config file at startup
* revert(build): changes were made to docker
* fix(docker): send variables correctly to the entrypoint
* test different conf file approach
* fix(env): add RUN_TEST env variable
* ref: use previous approach
* fix(color): use environment variable
* fix(resources): use our normal machine size
* fix(ci): double CPU and RAM for full sync test
* fix(test): check for zebrad test output in the correct order
The mempool is only activated once, so we must check for that log first.
After mempool activation, the stop regex is logged at least once.
(It might be logged before as well, but we can't rely on that.)
When checking that the mempool didn't activate,
wait for the `zebrad` command to exit,
then check the entire log.
* fix(ci): run full sync test with full compiler optimisations
* fix(tests): reintroduce tests and run full sync on approval
* fix(tests): reduce the changelog
Co-authored-by: teor <teor@riseup.net>
* fix(ci): update CI job path triggers (#3692)
* ci(test): re-run tests when snapshot data changes
* fix(ci): rebuild state when disk format changes
* fix(ci): rebuild rust docs when code or dependencies change
* doc(ci): explain why we run jobs when files change
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix(build): use the right multistage target (#3700)
* fix(review): only assign one reviewer to general Rust reviews (#3708)
If we assign two teams, GitHub assigns two reviewers.
* fix(ci): change the color-eyre ignore to a tracing-subscriber ignore
* fix(ci): ignore duplicate darling dependencies
* doc(ci): remove an alternative resolution doc
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* fix: check TEST_FAKE_ACTIVATION_HEIGHTS at runtime
* fix(tests): add TEST_FAKE_ACTIVATION_HEIGHTS variable
This variable ensures the test is activated in the `test-fake-activation-heights` step
* fix(docker): do not run specific tests by default in entrypoint.sh
* fix(test): remove extra TEST_FULL_SYNC argument
* imp(timeout): wait for an average build time
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* test(db): snapshot finalized tips
* test(db): snapshot block heights and hashes
* test(db): snapshot block data
* test(db): snapshot transaction hashes and data
* test(db): assert that blocks and transactions are in chain order
* clippy(db/test): cargo clippy --fix
* clippy(db/test): replace &Vec with &[_]
* refactor(state): split database access into modules by Zebra types
Also split the genesis block check from the genesis note commitment trees.
* test(db): snapshot column family names
* fix(db): assert that the default column family is empty on open and close
* feat(test): apply `cargo insta` settings to all tests
* doc(db/test): improve test docs
* test(db): snapshot column family data for the empty state
* refactor(db/test): split out the raw database snapshot
* test(db): snapshot raw database for blocks 0-2
* test(db): initial serialized snapshot data for raw database
* test(db): tweak snapshot file names
* test(db): rename snapshots for consistency
* test(db): store empty column families in a single snapshot
* test(db): simplify snapshot files by combining empty snapshots
* doc(db/test): put comment in a better place
* refactor(db): fastmod assert_default_is_empty assert_default_cf_is_empty
* doc(test): explain when insta settings are needed
* fix(state/test): use the network to initialize the state
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix(state): mark DiskWriteBatch as must_use
* doc(state): add TODOs for moving database reads to blocking threads
* doc(state): minor comment tweaks
* refactor(state): split write batch into block, transactions, chain history
* refactor(state): split out a genesis block write method
* refactor(state): just use the empty note commitment trees directly
* refactor(state): split transaction writes into transparent, nullifiers, trees
And change DiskWriteBatch methods to take `&mut self`.
* refactor(state): split chain value pool writes out of history writes
* refactor(state): combine note commitment trees into an agrument struct
* refactor(state): split history and note commitment updates
* refactor(state): calculate current tip height and remove that argument
* refactor(state): move disk_db reads to a new zebra_db module
* refactor(state): make finalized value pool method names consistent
* refactor(state): split database writes into the zebra_db module
* refactor(state): move the block batch method to DiskWriteBatch
* refactor(state): actually add the zebra_db module
Unfortunately, I've lost the interim changes to this file,
so this commit might be the only one that compiles.
* refactor(state): add a newly created file to the cached state CI job
* refactor(state): move RocksDB-specific initialization to a new module
* refactor(state): move RocksDB-specific shutdown to a new module
* refactor(state): temporarily allow RocksDB-specific reads and writes, without a new module
Unlike the last few commits, this one actually compiles.
* refactor(state): add a DiskWriteBatch wrapper for RocksDB writes
* refactor(state): move finalized state test methods to a test module
Also:
- upgrades to rlimit 0.7.0
- updates types to match the breaking changes in rlimit
- deletes a manual implementation that was similar to `increase_nofile_limit`,
but not as good on macOS and some BSDs
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* docs: document second part of consensus rules from 7.6 Block Header Encoding and Consensus
* docs: explain that the finalSaplingRoot check is not needed since we checkpoint on Canopy
* refactor(state): split the disk_format module
* refactor(ci): add the new disk_db file to the state CI list
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(network): split synthetic NotFoundRegistry from message NotFoundResponse
* docs(network): Improve `notfound` message documentation
* refactor(network): Rename MustUseOneshotSender to MustUseClientResponseSender
```
fastmod MustUseOneshotSender MustUseClientResponseSender zebra*
```
* docs(network): fix a comment typo
* refactor(network): remove generics from MustUseClientResponseSender
* refactor(network): add an inventory collector to Client, but don't use it yet
* feat(network): register missing peer responses as missing inventory
We register this missing inventory based on peer responses,
or connection errors or timeouts.
Inbound message inventory tracking requires peers to send `notfound` messages.
But `zcashd` skips `notfound` for blocks, so we can't rely on peer messages.
This missing inventory tracking works regardless of peer `notfound` messages.
* refactor(network): rename ResponseStatus to InventoryResponse
```sh
fastmod ResponseStatus InventoryResponse zebra*
```
* refactor(network): rename InventoryStatus::inner() to to_inner()
* fix(network): remove a redundant runtime.enter() in a test
* doc(network): the exact time used to filter outbound peers doesn't matter
* fix(network): handle block requests slightly more efficiently
* doc(network): fix a typo
* fmt(network): `cargo fmt` after rename ResponseStatus to InventoryResponse
* doc(test): clarify some test comments
* test(network): test synthetic notfound from connection errors and peer inventory routing
* test(network): improve inbound test diagnostics
* feat(network): add a proptest-impl feature to zebra-network
* feat(network): add a test-only connect_isolated_with_inbound function
* test(network): allow a response on the isolated peer test connection
* test(network): fix failures in test synthetic notfound
* test(network): Simplify SharedPeerError test assertions
* test(network): test synthetic notfound from partially successful requests
* test(network): MissingInventoryCollector ignores local NotFoundRegistry errors
* fix(network): decrease the inventory rotation interval
This stops us waiting 3-4 sync resets (4 minutes) before we retry a missing block.
Now we wait 1-2 sync resets (2 minutes), which is still a reasonable rate limit.
This should speed up syncing near the tip, and on testnet.
* fmt(network): cargo fmt --all
* cleanup(network): remove unnecessary allow(dead_code)
* cleanup(network): stop importing the whole sync module into tests
* doc(network): clarify syncer inventory retry constraint
* doc(network): add a TODO for a fix to ensure API behaviour remains consistent
* doc(network): fix a function doc typo
* doc(network): clarify how we handle peers that don't send `notfound`
* docs(network): clarify a test comment
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Remove redundant documentation
The documentation was exactly the same as the documentation from the
trait.
* Calculate a mock time block delta for tests
Simulate a block being added to the chain with a random block time based
on the previous block time and the target spacing time.
* Add a `time` field to `ChainTipBlock`
Store the block time so that it's ready for a future chain that allows
obtaining the chain tip's block time.
* Add `ChainTip::best_tip_block_time` method
Allow obtaining the bes chain tip's block time.
* Add method to obtain both height and block time
Prevent any data races by returning both values so that they refer to
the same chain tip.
* Add `NetworkUpgrade::all_target_spacings` method
Returns all the target spacings defined for a network.
* Create a `NetworkChainTipEstimator` helper type
Isolate the code to calculate the height estimation in a new type, so
that it's easier to understand and doesn't decrease the readability of
the `chain_tip.rs` file.
* Add `ChainTip::estimate_network_chain_tip_height`
This is more of an extension method than a trait method. It uses the
`NetworkChainTipHeightEstimator` to actually perform the estimation, but
obtains the initial information from the current best chain tip.
* Fix typo in documentation
There was an extra closing bracket in the summary line.
* Refactor `MockChainTipSender` into a separate type
Prepare to allow mocking the block time of the best tip as well as the
block height.
* Allow sending mock best tip block times
Add a separate `watch` channel to send the best tip block times from a
`MockChainTipSender` to a `MockChainTip`.
The `best_tip_height_and_block_time` implementation will only return a
value if there's a height and a block time value for the best tip.
* Fix off-by-one height estimation error
Use Euclidean division to force the division result to round down
instead of rounding towards zero. This fixes an off-by-one error when
estimating a height that is lower than the current height, because the
fractionary result was being discarded, and it should have forced the
height to go one block back.
* Fix panics on local times very far in the past
Detect situations that might cause the block height estimate to
underflow, and return the genesis height instead.
* Fix another off-by-one height estimation error
The implementation of `chrono::Duration::num_seconds` adds one to the
number of seconds if it's negative. This breaks the division
calculation, so it has to be compensated for.
* Test network chain tip height estimation
Generate pairs of block heights and check that it's possible to estimate
the larger height from the smaller height and a displaced time
difference.
* Support large block heights
* Document consensus rules referring to expiry heights
* Refactor the docs
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix the formatting of an error message
* refactor: Simplify coinbase expiry code so the consensus rule is clear (#3408)
* Fix some outdated TODO comments
* refactor(coinbase expiry): Simplify the code so consensus rule is clear
* Fix the formatting of an error message
* Remove a redundant comment
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* Check the max expiry height at parse time
* Test that 2^31 - 1 is the last valid height
* Add tests for nExpiryHeight
* Add tests for expiry heights of V4 transactions
* Add tests for V5 transactions
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feat(network): send notfound messages to the inventory registry
* refactor(network): move the inventory filter into an async function
* feat(network): avoid routing requests to peers that are missing inventory
* test(network): advertised routing is independent of numeric address value
* test(network): peer set routes requests to peers not missing that inventory
* test(network): peer set fails requests if all ready peers are missing that inventory
* fix(clippy): needless-borrow in the peer set
* fix(lint): remove redundant trailing commas in macro calls
There is no clippy lint for this, maybe because some macros
are sensitive to trailing commas.
(But not the ones changed in this commit.)
* test(network): check the exact number of inventory peers
* doc(network): explain why we ignore inventory send failures
* docs(network): explain why a channel error is ignored
* feat(log): log the state tip height as part of sync progress logs
* fix(log): downgrade some verbose state logs to debug
* feat(log): log successful gossiped block verification at info level
These logs help us diagnose slow progress near the tip.
There won't be very many of these logs,
because they only happen near the tip.
* fix(log): spawn top-level tasks within the global Zebra tracing span
* fix(log): spawn blocking top-level tasks within the global Zebra tracing span
Co-authored-by: teor <teor@riseup.net>
* lint: enable more clippy checks for bug-prone code
* fix(lint): stop denying lints, to avoid being excluded from Crater
Also categorise lints.
* lint: add some lints to the TODO list
* refactor(arithmetic): partial fixes for some integer arithmetic lints
* Document some weird lint behaviour
* Updating zebra-test to v1.0.0-beta.4
* Updating tower-fallback to v0.2.16
* Updating tower-batch to v0.2.20
* Updating zebra-chain to v1.0.0-beta.4
* Updating zebra-script to v1.0.0-beta.4
* Updating zebra-network to v1.0.0-beta.4
* Updating zebra-state to v1.0.0-beta.4
* Updating zebra-consensus to v1.0.0-beta.4
* Updating zebra-utils to v1.0.0-beta.4
* Updating zebrad to v1.0.0-beta.4
* Update Cargo.lock with updated crate versions
* Add section for 1.0.0-beta.4 in the CHANGELOG
* Update CHANGELOG.md
Co-authored-by: teor <teor@riseup.net>
* Update README.md
Co-authored-by: teor <teor@riseup.net>
* Update book/src/user/install.md
Co-authored-by: teor <teor@riseup.net>
* Update release description to mention Rust 2021
Update the changelog to mention that all crates now use Rust 2021
Edition.
* Elaborate on Section 3.6 documentation entry
Add the title of the section and mention that it's a section of the
Zcash protocol specification.
* Make changelog entries consistent
Use the same format for entries related to consensus rule documentation.
Co-authored-by: Marek <mail@marek.onl>
* Update README about validated consensus rules
Zebra now validates all documented consensus rules.
* Add changelog entry for newly merged PR
Describe the security fix in the changelog.
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
* cancel background database tasks in `FinalizedState` destructor
* use `shutdown_timeout()`
* Log info-level messages while waiting for background tasks to shut down
* Cancel background tasks during debug_stop_at_height shutdown
This commit moves the database shutdown code into a common function.
* Create a constant for the tokio timeout
* Add a test script for Zebra shutdown errors
* Increase the shutdown timeout to 20 seconds for slower machines
* add title to building zebra
* use imported duration
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Move standard lints into .cargo/config.toml
* Ignore "wrong self convention" in a futures-based trait
This lint might only trigger on beta or nightly at the moment.
* Warn if future incompatibile code is added to Zebra
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Avoid sequential borrows in `LatestChainTip`
Calling `watch::Receiver::borrow` more than once in the same scope can
cause a deadlock. The instrumented methods were calling `borrow` twice
to record instrumented fields.
This refactors things to ensure `borrow` is only called once to record
the fields and perform any actions with the chain tip block.
* Remove `borrow()` calls in `ChainTipChange`
Refactor to use a `LatestChainTip` instance instead, which safely
protects the internal `watch::Receiver` so that it is not borrowed more
than once in the same scope.
* Add a paragraph to the Asynchronous guide
Warn against using two borrow guards in the same scope, and describe why
that can lead to a deadlock.
* Use `testdir()` instead of `TempDir::new()`
Reduce repeated code and make it easier to change from using `tempdir`
to use `tempfile` instead.
* Replace `tempdir` with `tempfile` in `zebrad`
Use `tempfile`'s `TempDir` instead.
* Use `tempdir()` instead of `TempDir::new()`
Reduce repeated code and make it easier to upgrade to `tempfile`.
* Use `tempfile` instead of `tempdir`
Replace obsoleted `tempdir` dependency with `tempfile`.
* Use `tempfile` instead of `tempdir`
Replace obsoleted `tempdir` dependency with `tempfile`.
* Update `Cargo.lock`
Update it now that `tempdir` has been replaced with `tempfile`.
* Remove `tempdir` from `deny.toml` exceptions
Ban duplicate versions of the `tempdir` dependency.
* Remove `inferno` from `deny.toml` exceptions
It apparently isn't needed anymore.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Do prelim checking of Sprout anchors in non-finalized state
Does not check intra-transaction interstitial states yet
* Populate sprout anchors to allow other state tests to pass
* Preliminary interstitial sprout note commitment tree anchor checks implementation
* Make sure only prior anchors are checked in the same transaction
* Add tests
* Refactor a comment
* Refactor rustdoc
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
* Use the first `JoinSplit`s from mainnet
* Print debug messages
* Use correct blocks for the tests
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Zebra's latest beta continues implementing zero-knowledge proof and note commitment tree validation. In this release, we have finished implementing transaction header, transaction amount, and Zebra-specific NU5 validation. (NU5 mainnet validation is waiting on an `orchard` crate update, and some consensus parameter updates.)
We also fix a number of security issues that could pose a local denial of service risk, or make it easier for an attacker to make a node follow a false chain.
As of this release, Zebra will automatically download and cache the Sprout and Sapling Groth16 circuit parameters. The cache uses around 1 GB of disk space. These cached parameters are shared across all Zebra and `zcashd` instances run by the same user.
See CHANGELOG.md for the full list of changes in this release.
* Add Transaction::sprout_joinsplits()
* Add Anchor variants to ValidateContextError
* Make Chain anchor collections pub(crate)
* tracing::instrument several methods in state
* Add contains_*_anchors methods to FinalizedState
* Add check::anchors module and function
* Verify that anchors_refer_to_earlier_treestates in when updating chains in non-finalized state
* Update zebra-state/src/service/check/anchors.rs
Co-authored-by: teor <teor@riseup.net>
* Add anchors() to sapling::ShieldedData
* Add sapling_anchors() to Transaction
* Use Transaction::sapling_anchors() in the anchors_refer_to_earlier_treestates() check
* Whoops, itertools
* Add a comment for improvement
Co-authored-by: teor <teor@riseup.net>
* Add & use a cfg(test) method on FinalizedState to prep test state with anchors to allow other tests to pass contextual checks
* Allow test nullifier checks to pass by populating anchor sets, allowing test anchor checks to pass
* Add mainnet block 419202 and its sapling note commitment tree root to test vectors
* Test sapling anchor verification using the first few Sapling blocks data
* Correct comment
* assert_eq instead of assert(bool)
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Update zebra-state/src/service/non_finalized_state.rs
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Create a `LockTime::unlocked` helper constructor
Returns a `LockTime` that is unlocked at the genesis block.
* Return `Option<LockTime>` from `lock_time` method
Prepare to return `None` for when a transaction has its lock time
disabled.
* Return `None` instead of zero `LockTime`
Because a zero lock time means that the transaction was unlocked at the
genesis block, so it was never actually locked.
* Rephrase zero lock time check comment
Clarify that the check is not redundant, and is necessary for the
genesis transaction.
Co-authored-by: teor <teor@riseup.net>
* Add a `transparent::Input::sequence` getter method
Retrieve a transparent input's sequence number.
* Check if lock time is enabled by a sequence number
Validate the consensus rule that the lock time is only enabled if at
least one transparent input has a value different from `u32::MAX` as its
sequence number.
* Add more Zcash specific details to comment
Explain the Zcash specific lock time behaviors.
Co-authored-by: teor <teor@riseup.net>
* Add `time` field to `Request::Block` variant
The block time to use to check if the transaction was unlocked and
allowed to be included in the block.
* Add `Request::block_time` getter
Returns the block time for the block that owns the transaction being
validated or the current time plus a tolerance for mempool transactions.
* Validate transaction lock times
If they are enabled by a transaction's transparent input sequence
numbers, make sure that they are in the past.
* Add comments with consensus rule parts
Make it easier to map what part of the consensus rule each match arm is
responsible for.
Co-authored-by: teor <teor@riseup.net>
* Upgrade aes and fpe
* Upgrade bellman, bls12_381, jubjub to latest
* Upgrade x25519-dalek to 1.2.0 and curve25519-dalek to 3.2.0 in the Cargo.lock
* Skip outdated hdrhistogram rather than its dependencies
Co-authored-by: teor <teor@riseup.net>
* Cleanup a function that calls zcash_script
* Remove zebra_test::prelude macros that conflict with the Rust prelude
* Add sigops count support to zebra-script
* Check MAX_BLOCK_SIGOPS in the block verifier
* Test MAX_BLOCK_SIGOPS on generated and historic blocks
* Add SAFETY comments for all unsafe zebra-script code
* Explain where the consensus rule comes from
* Remove unused pretty_assertions dependency
* Allow large test block generation functions with the proptest-impl feature
* Replace `as` with `try_into` for integer conversions in unsafe code
* Expand SAFETY comments
* Update `tower` to version `0.4.9`
Update to latest version to add support for Tokio version 1.
* Replace usage of `ServiceExt::ready_and`
It was deprecated in favor of `ServiceExt::ready`.
* Update Tokio dependency to version `1.13.0`
This will break the build because the code isn't ready for the update,
but future commits will fix the issues.
* Replace import of `tokio::stream::StreamExt`
Use `futures::stream::StreamExt` instead, because newer versions of
Tokio don't have the `stream` feature.
* Use `IntervalStream` in `zebra-network`
In newer versions of Tokio `Interval` doesn't implement `Stream`, so the
wrapper types from `tokio-stream` have to be used instead.
* Use `IntervalStream` in `inventory_registry`
In newer versions of Tokio the `Interval` type doesn't implement
`Stream`, so `tokio_stream::wrappers::IntervalStream` has to be used
instead.
* Use `BroadcastStream` in `inventory_registry`
In newer versions of Tokio `broadcast::Receiver` doesn't implement
`Stream`, so `tokio_stream::wrappers::BroadcastStream` instead. This
also requires changing the error type that is used.
* Handle `Semaphore::acquire` error in `tower-batch`
Newer versions of Tokio can return an error if the semaphore is closed.
This shouldn't happen in `tower-batch` because the semaphore is never
closed.
* Handle `Semaphore::acquire` error in `zebrad` test
On newer versions of Tokio `Semaphore::acquire` can return an error if
the semaphore is closed. This shouldn't happen in the test because the
semaphore is never closed.
* Update some `zebra-network` dependencies
Use versions compatible with Tokio version 1.
* Upgrade Hyper to version 0.14
Use a version that supports Tokio version 1.
* Update `metrics` dependency to version 0.17
And also update the `metrics-exporter-prometheus` to version 0.6.1.
These updates are to make sure Tokio 1 is supported.
* Use `f64` as the histogram data type
`u64` isn't supported as the histogram data type in newer versions of
`metrics`.
* Update the initialization of the metrics component
Make it compatible with the new version of `metrics`.
* Simplify build version counter
Remove all constants and use the new `metrics::incement_counter!` macro.
* Change metrics output line to match on
The snapshot string isn't included in the newer version of
`metrics-exporter-prometheus`.
* Update `sentry` to version 0.23.0
Use a version compatible with Tokio version 1.
* Remove usage of `TracingIntegration`
This seems to not be available from `sentry-tracing` anymore, so it
needs to be replaced.
* Add sentry layer to tracing initialization
This seems like the replacement for `TracingIntegration`.
* Remove unnecessary conversion
Suggested by a Clippy lint.
* Update Cargo lock file
Apply all of the updates to dependencies.
* Ban duplicate tokio dependencies
Also ban git sources for tokio dependencies.
* Stop allowing sentry-tracing git repository in `deny.toml`
* Allow remaining duplicates after the tokio upgrade
* Use C: drive for CI build output on Windows
GitHub Actions uses a Windows image with two disk drives, and the
default D: drive is smaller than the C: drive. Zebra currently uses a
lot of space to build, so it has to use the C: drive to avoid CI build
failures because of insufficient space.
Co-authored-by: teor <teor@riseup.net>
* Try simulating a chain growth
* Adjust the transaction expiry height
The mempool evicts expired transactions. When working with mocked data,
appending a new block typically clears the mempool because transactions become
expired. For this reason, the expiry height of each transactions is adjusted so
that it is greater than the new chain tip's height.
* Refactor the code so that it works with `VerifiedUnminedTx`
* Fix a typo
* Fix clippy warnings
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
There are a lot of these messages when Zebra starts up.
They might be slowing down CI and causing timeouts.
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Increment the crates that have new commits since the last version
* Increment the crates that depend on crates that have changed
* Increment the version of `zebra-script`
* Use the `zebrad` version in the `zebra-network` user agent string
* Use the `v1.0.0-alpha.19` git tag in `README.md`
* Copy the draft changelog into `CHANGELOG.md`
* Delete bumps
* Update CHANGELOG.md
Co-authored-by: teor <teor@riseup.net>
* Add newly merged PRs
Co-authored-by: teor <teor@riseup.net>
* Increase the restart test timeout to 10 seconds
It shouldn't take this long.
But maybe the CI VMs are under a lot of load?
* Add extensive logging to diagnose CI state reload failures
* Check for state errors before updating metrics
Previously, the metrics would be updated for some rejected blocks.
* Clarify and expand block verification metrics
Rename checkpoint-specific metrics to clarify their purpose.
Add metrics for:
- finalized blocks on disk
- blocks verified using the full block verifier
(this metric was previously incorrectly called `zcash_chain_verified_block_height`)
* Update dashboard metric names
Also:
- add some extra block height metrics
- fix a dashboard name
* Add exact block heights to Grafana dashboards
* Add a missing comment
* grafana: use 0 decimals for metrics
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* grafana: show the entire height instead of abbreviated
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* grafana: show the entire height instead of abbreviated
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Fix typo in metric name
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Move height gauges to the state, so they are correct
If we update height gauges in futures, they can execute out of order,
so the metrics can be incorrect.
Instead:
- move the height gauges to the state, and update them based on the best tip
- move the verified block counts to the state
- continue to include all verified blocks on all non-finalized chains
(not just the best chain)
* Show exact checkpoint heights in the dashboard
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Implement a task that gossips verified block hashes
* Log an info message for block broadcasts
* Simplify the gossip task
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Re-use the old tip change if there is no new tip change
Also improve the comments.
* Add an assertion message
* Rename task join handles and futures in start method
* Add a dedicated BlockGossipError type
This type helps distinguish between syncer and state errors.
* Test that committed blocks are gossiped to peers
Also do a minor type cleanup on the existing test code,
replacing `Option<Vec<_>>` with `Vec<_>`.
* Formatting
* Remove excess newlines
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Clear the initial gossiped blocks during test setup
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Update versions for zebra v1.0.0-alpha.18 release
* WIP: Initial PR list
* Remove uninteresting version bumps from CHANGELOG
* Categorise and group PRs in CHANGELOG, removing uninteresting PRs
* Further refine and categorise changelog entries
* Fix tag url
* Final changes to CHANGELOG
* Add a changelog description
* Spacing
* Clarify and fix changelog PR descriptions
* Add PRs that are about to be merged
* More slight clarifications
* Spacing
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Check return value of zcash_script_new_precomputed_tx
* Set the NU5 testnet activation height to 1_590_000
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Update Nu5 constants to new values
* Update ZIP-244 test vectors for new branch ID
* Squashed commit of the following:
commit bdb120a249
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 11:54:01 2021 -0400
Use pallas::Base::from_str_vartime() in sinsemilla tests
commit e99fa49258
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 11:45:24 2021 -0400
Compiles
commit a520018114
Author: Deirdre Connolly <durumcrustulum@gmail.com>
Date: Tue Oct 5 10:15:17 2021 -0400
Incomplete upgrade of deps
* Squashed commit of the following:
commit 8d1b76ec5626517817c3a4d9f3950acc90a359df
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 04:02:26 2021 +0000
Update `zcash_script` to support V5 transactions
Use a newer version of `zcash_script` that has been updated to support
V5 transactions.
commit 371233628ae61e0c25d6ba8f31d9dba42823becb
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 03:06:20 2021 +0000
Update Zcash dependencies
Update some Zcash crates:
- `halo2`
- `incrementalmerkletree' (patch version)
- `orchard` (patch version)
- `zcash_history` (patch version)
- `zcash_note_encryption` (patch version)
- `zcash_primitives` (patch version)
And also update the `group` dependency so that the code remains
compatible.
commit de5cf1ec40c3fc08670fc971cdf3e65e13d9f4c7
Author: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Date: Tue Oct 5 03:04:13 2021 +0000
Update error message assertion
Use the updated message for the expected error variant.
* Update `zcash_script` to support V5 transactions
Use a newer version of `zcash_script` that has been updated to support
V5 transactions.
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
* Move mempool tests into `tests::vector` sub-module
Make it consistent with other test modules and prepare for adding
property tests.
* Reorder imports
Make it consistent with the general guidelines followed on other
modules.
* Export `ChainTipBlock` and `ChainTipSender`
Allow these types to be used in other crates for testing purposes.
* Derive `Arbitrary` for `ChainTipBlock`
Make it easy to generate random `ChainTipBlock`s for usage in property
tests.
* Refactor to move test methods into `tests` module
Reduce the repeated test configuration attributes and make it easier to
see what is test specific and what is part of the general
implementation.
* Add a `Mempool::dummy_call` test helper method
Performs a dummy call just so that `poll_ready` gets called.
* Use `dummy_call` in existing tests
Replace the custom dummy requests with the helper method.
* Test if the mempool is cleared on chain reset
A chain reset should force the mempool storage to be cleared so that
transaction verification can restart using the new chain tip.
* Test if mempool is cleared on syncer restart
If the block synchronizer falls behind and then starts catching up
again, the mempool should be disabled and therefore the storage should
be cleared.
* Instrument chain tip methods
* Expand tests to cover last_tip_change and multiple change checks
* Expand tests to cover Grow as well as Reset
* Support sync and async methods on the same ChainTipChange
* Add a Tokio 1.0 TODO
* Clarify a comment
* Manual rustfmt inside a proptest
* Remove tracing clones, and instrument ChainTipSender::new
* Add the tokio issue number to a TODO comment
teor
Marek
Dimitris Apostolou
Janito Vaqueiro Ferreira Filho
dependabot[bot]
Alfredo Garcia
Conrado Gouvea
Deirdre Connolly
Pili Guerra