* fix(actions): use a specific shortening length for SHAs
The rlespinasse/github-slug-action now works without checking out the code, reduce time and improving security with following actions.
This requires to specify the GITHUB_SHA_SHORT variable length, as git uses 8 by default, but docker uses 7 by default.
* fix(actions): target correct rlespinasse/github-slug-action version
* fix(actions): just use major version
* fix(actions): github-slug-action is not being correctly referenced
* Simplify the database design using prefix iterators
* Fix typos and missed changes
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Marek <mail@marek.onl>
* refactor(ci): use improved OIDC authentication
* fix(ci): standardize OIDC on all required jobs
* fix: wrong indentation
* fix(ci): remove non existing depency in clean job
* Add round-trip test for OutputLocation disk serialization
* Make the transaction snapshot tests more accurate
Previously, we were showing the genesis transaction hash at height 0, index 0.
But its hash is actually not stored by location in the database,
because the genesis transaction is skipped due to a consensus rule.
* Update the transaction snapshot data
* Add history tree snapshot tests
At the current test heights, the history trees are all empty.
* Add the history tree snapshot data
* Update comments
* Simplify snapshot test code
* Make some serde::Serialize impls test-only, so we can hex-encode them
This should also speed up release compilation a bit.
* Add snapshot test code for UTXOs
* Add snapshot data for UTXOs
* Refactor to split `ChainSync::sync` method in two
Replace the use of loop labels and `continue` for control flow, and use
early return from a separate method instead. This also allows removing
the `started_once` flag.
* Refactor to create `handle_block_response` helper
Reduce duplicate code and make the main synchronization methods a little
more concise to improve readability.
* Only cancel downloads in case of error
Leave active downloads running if the tips have been exhausted, because
it could have reached the chain tip.
* Split out ReadRequest and ReadResponse state service enums
* Simplify RPC test vectors
* Split state requests into Request and ReadRequest
* Make zebra-rpc use the new state ReadRequest
* Add request metrics to ReadStateService
Add a field to distinguish StateService and ReadStateService metrics.
Add missing StateService request metrics.
* Refactor state transaction lookup so it can be shared between services
* Implement ReadState Request::Transaction
* Test if mandatory checkpoint is after grace period
Check that the mandatory checkpoint height is after the ZIP-212 grace
period for both mainnet and testnet, because Zebra is currently unable
to fully validate the blocks during that grace period.
* Update mandatory checkpoint height
Require it to be after the ZIP-212 grace period so that the blocks
during the grace period are validated by the checkpoints because Zebra
can't currently fully validate blocks during that period.
* Revert "Drop the Chain watch channel lock before accessing the finalized state"
This reverts commit 8870944d13.
* Add a WatchReceiver wrapper that always clones the borrowed watch data
This avoids deadlocks, by holding the read lock for as short a time as possible.
* Drop the shared Arc<Chain>s as quickly as possible
This reduces memory usage.
* Make read::block more flexible, by accepting any AsRef<Chain>
* Make the block method docs consistent
* Avoid livelocks by explicitly dropping the borrow after the clone
* Add the ChainTip and Network to RpcImpl
* Add the new RpcImpl fields to the tests
* Simplify RPC version field using generics
* Temporarily allow unused struct fields
* Use the read-only state service for RPCs
* Refactor non-finalized block lookup into Chain
* Implement the read-only state block request
* Drop the Chain watch channel lock before accessing the finalized state
* Query Sapling & Orchard trees by height in the finalized state
* Add Sapling & Orchard trees to the non-finalized state
* Add a TODO about concurrent read-only access to Sprout tree
Co-authored-by: teor <teor@riseup.net>
* Update the database format version
* Keep only the most recent Sprout tree in the database
* Check that the database returns empty trees for the genesis block
* Assert that the database returns the highest trees
* Document how to update insta snapshots
* Add note commitment tree insta snapshot tests
* Add comments about cached tree roots in snapshots
* Add snapshot data for sapling and orchard trees
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* feat(rpc): implement `getrawmempool` rpc method
* tests(rpc): start implementing a test
* Return a `Vec<String>` from `get_raw_mempool`
Ensure the returned type is serialized as an array of hexadecimal
strings.
* Make `getrawmempool` test a property test
Test it with random sets of transaction IDs.
* Fix code formatting inside `proptest!`
Run `rustfmt` inside the `proptest!` block.
* tests(rpc): fix lightwalletd test
* fix(rpc): remove non used return type
* tests(rpc): remove assert
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Allow extra time for lightwalletd integration tests
* Wait for a successful block ingestion in lightwalletd integration
* Check for any ingestor block, not just block 1
* Add state service module docs and cleanup
* Move and add finalized state methods
* Add chain and non-finalized state methods
* Cleanup methods and imports
* Create a ReadStateService type
* Add a stub service implementation
* Add a TODO
* Update ReadStateService request stubs with RPC names and tickets
* Documentation updates
* Make RPC State generic bounds accept a buffered state and a read-only state
* Doc updates
* Add missing proptest-impl feature in RPC dev dependencies
* feature(rpc): start adding a `getblock` method
* fix(rpc): replace oneshot
* fix(rpc): replace a panic with error
* fix(rpc): fix test
* feature(rpc): add hex to response
* refactor(rpc): use generic instead of alias
* docs(rpc): improve docs for getblock method
* test(rpc): add a test for getblock method
* deps(rpc): remove non needed tower features
Co-authored-by: teor <teor@riseup.net>
* docs(rpc): add a note to getblock doc
* refactor(rpc): replace alias
* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format
* tests(rpc): add network argument to `populated_state()`
* refactor(rpc): use an error for state service readiness
* fix(rpc): add parameter
* fix(rpc): clippy
* nit(rpc): remove new line from imports
* fix(rpc): remove commented code
* fix(rpc): simplify error
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use a `SerializedBlock` type to help serializing blocks (#3725)
* Create a `SerializedBlock` helper type
Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.
* Use `into_iter` instead of `iter`
There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.
* Use `SerializedBlock` in `GetBlock`
Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.
The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.
* feature(rpc): add suggestions from code review
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* tests(rpc): make sure mempool has no requests in get_block test
* fix(rpc): change height argument type in getblock method
* fix(rpc): rustfmt
* fix(rpc): replace panic
* fix(rpc): change getblock response
* fix(rpc): fix lightwalletd test
* tests(rpc): add a getblock error test
* fix(rpc): try another regex
* feature(rpc): add `getbestblockhash` RPC method
* feature(rpc): Add a `pub struct SerializedBlockHash` type
* tests(rpc): add a unit test for `getbestblockhash` method
* tests(rpc): make sure no requests are sent to mempool in getbestblockhash test
* tests(rpc): refactor check
Co-authored-by: teor <teor@riseup.net>
* fix(rpc): fixes after rebase
* refactor(rpc): refactor `GetBestBlockHash`
* fix(rpc): unused variables
Co-authored-by: teor <teor@riseup.net>
* docs(rpc): update
* fix(rpc): add panic
* fix(rpc): fix panic
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Move the legacy chain check to the `check` module
And move `populated_state` to the `arbitrary` module.
* Cleanup imports
* Document the state service struct
* Split state block iter into its own module
* Prepare the finalized state for read-only state
* Add a forced shutdown mode, used in test code before forced exits
* Document the small database drop race condition window
* Move the legacy chain check to the `check` module
And move `populated_state` to the `arbitrary` module.
* Cleanup imports
* Document the state service struct
* Split state block iter into its own module
* Drop chain if it becomes invalid
Avoid returning a chain that shouldn't be used again by having the
method take ownership of `self` and dropping it on error.
* Update documentation and comments
Restore them to be closer to how they were on `main`.
* fix(shielded): use RwLock for note commitment tree root caches
* Prepare the non-finalized state for read-only state
* fix(state): performance tweak for chain forks
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* feature(rpc): start adding a `getblock` method
* fix(rpc): replace oneshot
* fix(rpc): replace a panic with error
* fix(rpc): fix test
* feature(rpc): add hex to response
* refactor(rpc): use generic instead of alias
* docs(rpc): improve docs for getblock method
* test(rpc): add a test for getblock method
* deps(rpc): remove non needed tower features
Co-authored-by: teor <teor@riseup.net>
* docs(rpc): add a note to getblock doc
* refactor(rpc): replace alias
* fix(rpc): use `zcash_serialize_to_vec()` instead of logging format
* tests(rpc): add network argument to `populated_state()`
* refactor(rpc): use an error for state service readiness
* fix(rpc): add parameter
* fix(rpc): clippy
* nit(rpc): remove new line from imports
* fix(rpc): remove commented code
* fix(rpc): simplify error
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Use a `SerializedBlock` type to help serializing blocks (#3725)
* Create a `SerializedBlock` helper type
Create a type that can be used as a byte slice, but is guaranteed to
represent a valid block.
* Use `into_iter` instead of `iter`
There's no need to borrow the elements, they can be moved out directly.
This will be necessary because `&Arc<T>` doesn't implement `Borrow<T>`,
so a `SerializedBlock` can't be built directly from an `&Arc<Block>`.
* Use `SerializedBlock` in `GetBlock`
Make the type stricter to avoid storing possibly invalid values. The
bytes are still serialized as a hexadecimal string, through the usage of
`hex`.
The `serde::Deserialize` can't be derived because `hex` requires the
type to also implement `FromHex`.
* feature(rpc): add suggestions from code review
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* tests(rpc): make sure mempool has no requests in get_block test
* fix(rpc): change height argument type in getblock method
* fix(rpc): rustfmt
* fix(rpc): replace panic
* fix(rpc): change getblock response
* fix(rpc): fix lightwalletd test
* tests(rpc): add a getblock error test
* fix(rpc): try another regex
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
GitHub's GHA cache gets invalidated at 10Gb, which is very easy to hit when we're building multiple times a day with several commits.
Instead use the registry, which won't get invalidated until a change is identified in the build process.
* refactor(test/block): rename large single transaction function
```sh
fastmod single_transaction_block single_transaction_block_many_inputs
```
* rustfmt
* test(block): add a test block with many transparent outputs
* doc(db): explain why we can't just get the UTXOs right before they are deleted
* refactor(db): split out a block data write method
* refactor(block): add a height argument to new_outputs
* test(db): add block and transaction round-trip tests
Including large blocks and transactions.
* test(db): fix large block serialization instability in the tests
* doc(block): add TODOs for generating correct blocks
* Make transparent output functions which take a height test-only
* make sure generated blocks are actually over/under-sized
* replace println!() with an error!() log
* refactor(test): reuse same GCP instance on a single PR
This also ensures the deployments are faster, and we only delete the instance when merging or closing the PR, instead of doing it on each push to the PR
* fix(deploy): add zone to updates
* fix: typo
* fix(ci): improve conditions for updates
* fix(deploy): delete old deployments instead of reusing it
* fix(deploy): keep delete command after run
* fix(deploy): always create an instance
* fix(deploy): delete disks on every delete command.
* imp(ci): use better id name
* Update .github/workflows/test.yml
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* imp: handle errors correctly on deletion
* fix: do not hide valid errors
* fix: edge case where the container is not ready yet
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Gustavo Valverde
teor
dependabot[bot]
Conrado Gouvea
Janito Vaqueiro Ferreira Filho
Alfredo Garcia
Marek
Dimitris Apostolou
Deirdre Connolly