* Update the reference Sapling treestate
Zebra's treestate serialization differs from `zcashd` in the following
way:
`zcashd` omits the serialization of empty trailing ommers, while Zebra
doesn't. This means that `zcashd` serializes the Sapling treestate for
height 419_201 as
019eb30778ddeea84c72e69e07a1689f3c8def3dc0a1939f0edcbe47279069d9310002000150715810d52caf35471d10feb487213fbd95ff209122225b7b65d27a7fb1a44d
Whereas Zebra serializes it as
019eb30778ddeea84c72e69e07a1689f3c8def3dc0a1939f0edcbe47279069d931001f000150715810d52caf35471d10feb487213fbd95ff209122225b7b65d27a7fb1a44d0000000000000000000000000000000000000000000000000000000000
Serialization Format
====================
The serialized treestate consists of optional, hex-encoded, 32-byte
hashes. If the hash is not present, it is serialized as byte 0, i.e.,
`0x00`. If the hash is present, it is prefixed by byte 1, i.e. `0x01`.
The first two hashes in the serialized treestate are the treestate's
left and right leaves. These are followed by the serialized length of
the vector of ommers. This length is serialized as 1, 3, 5, or 9 bytes.
If the length is less than 253, it is serialized as a single byte. The
length is then followed by the serialized ommers.
We can now parse the first string, produced by `zcashd`:
- `0119eb30778ddeea84c72e69e07a1689f3c8def3dc0a1939f0edcbe47279069d931`
is the serialized left leaf,
- `00` is the serialized right leaf,
- `02` is the serialized length of the vector of ommers,
- `00` is the serialized first ommer,
- `0150715810d52caf35471d10feb487213fbd95ff209122225b7b65d27a7fb1a44d`
is the serialized second ommer.
And the second one, produced by Zebra:
- `0119eb30778ddeea84c72e69e07a1689f3c8def3dc0a1939f0edcbe47279069d931`
is the serialized left leaf,
- `00` is the serialized right leaf,
- `1f` is the serialized length of the vector of ommers,
- `00` is the serialized first ommer,
- `0150715810d52caf35471d10feb487213fbd95ff209122225b7b65d27a7fb1a44d`
is the serialized second ommer
- `0000000000000000000000000000000000000000000000000000000000` are the
remaining 29 serialized ommers.
Note that both serializations represent the same treestate.
* Remove a new line char
A LWD test was expecting the `ZEBRA_TEST_LIGHTWALLETD` to be set, but this variable is needed for all LWD tests and not specifically for `lightwalletd_integration`.
We had to rename this variable on a buggy `elif` statement in our Docker entrypoint.
This was avoiding most LWD tests to run correctly.
* Adds getblockheader RPC method
* Updates snapshots, adds hash/height/next_block_hash fields to verbose response
* updates getblock snapshot
* updates getblockheader response type to hex-encode fields, adds ToHex impl for sapling::tree::Root, adds snapshot and vector tests for new RPC method, adds snapshots.
* rustfmt
* fixes snapshots, matches zcashd more closely
* fixes vectors test
* updates lwd failure messages (probably doesn't matter, but seems better to handle it now than risk debugging it later)
* fixes getblock_rpc test, fixes/reverses finalsaplingroot field byte-order.
* fixes vector test, addresses remaining differences with zcashd (except the `chainwork` field), updates snapshots, and avoids a possible panic when there's a chain reorg between state queries.
* Adds a comment noting that the `relative_to_network()` method was copied from zcashd
* Apply suggestions from code review
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Adds a parameter to `zebra_consensus::router::init()` for accepting a mempool setup argument, adds and uses an `init_test()` fn for passing a closed channel receiver in tests where no mempool service is needed in the transaction verifier.
* Adds a `mempool` argument to the transaction::Verifier constructor (and a `new_for_tests()` constructor for convenience)
* Removes `Clone` impl on `transaction::Verifier` to add mempool oneshot receiver, updates tests.
* Adds TODOs
* updates transaction verifier's poll_ready() method to setup the mempool service handle.
* Updates VerifiedSet struct used in mempool storage
* Updates mempool service and its `Storage` to use the updated `VerifiedSet` `transactions()` return type.
* updates `created_outputs` when inserting or removing a transaction from the mempool's verified set
* Adds a TODO, updates field docs
* Updates `spent_utxos()` to query the mempool for unspent outputs
* Adds `spent_mempool_outpoints` as a field on tx verifier mempool response
* Updates mempool `Downloads` to return the spent_mempool_outpoints from the tx verifier response
* Updates `Storage.insert()` to accept a list of spent mempool transaction outputs
* Adds transaction dependencies when inserting a tx in `VerifiedSet`
* polls mempool svc from tx verifier when a mempool tx that creates transparent outputs has been verified.
adds a TODO for adding a `pending_outputs` field to the mempool Storage
* Adds `pending_outputs` field on mempool Storage and responds to pending outputs requests when inserting new transactions into the mempool's verified set
* replaces `UnminedTxId` type with `transaction::Hash` in mempool's verified set
* prune pending outputs when rejecting and removing same effects.
* Remove dependent transactions from verified set when removing a tx
* updates tests
* appeases clippy.
* removes unused `len()` method
* fixes doc links
* Adds transaction dependencies to the `FullTransactions` response, let the caller handle it (required to avoid moving zip317 tx selection code to mempool)
* updates block template construction to avoid including transactions unless their dependencies have already been added.
* updates tests
* Replaces placeholder setup channel with one that sends the mempool svc to the tx verifier, adds a timeout layer, adds a TODO about a concurrency bug
* Use a single query to check for unspent outputs in the mempool
* Updates `getblocktemplate` method to consider dependencies when sorting transactions for the final template
* fixes clippy lints, removes unnecessary Option in UnspentOutput response variant
* renames type alias and method, adds a TODO to use iteration instead of recursion
* Adds mempool_removes_dependent_transactions() test
* Updates Storage and VerifiedSet clear() methods to clear pending_outputs, created_outputs, and transaction_dependencies, adds TODO to use iteration instead of recursion.
* removes outdated TODO
* Adds a TODO for reporting queued transaction verification results from the mempool from the poll_ready() method
* Adds `mempool_responds_to_await_output` test
* updates mempool_responds_to_await_output test
* Uses iteration instead of recursion in verified set's remove() method and zip317 mod's dependencies_depth() method
* Adds a mempool_request_with_mempool_output_is_accepted test for the transaction verifier
* Moves delay duration before polling the mempool to a constant, uses a shorter timeout for mempool output lookups, adds a `poll_count` to MockService, and updates `mempool_request_with_unmined_output_spends_is_accepted` to check that the transaction verifier polls the mempool after verifying a mempool transaction with transparent outputs
* adds long_poll_input_mempool_tx_ids_are_sorted test
* Adds a `excludes_tx_with_unselected_dependencies` test
* Updates a TODO
* moves `TransactionDependencies` struct to `zebra-node-services`
* Updates `FullTransactions` response variant's `transaction_dependencies` type
* updates zip317 transaction selection for block templates to include dependent transactions
* Moves and refactors zip317 tx selection test to its own module, adds an `unmined_transactions_in_blocks()` method on network
* Removes `unmined_transactions_in_blocks()` test utility fn from mempool Storage test module and replaces calls to it with calls to the new test method on Network
* Fixes spelling mistake
* Adds `includes_tx_with_selected_dependencies` test
* fixes zip317 block construction issue
* Fixes vectors test
* Update zebra-node-services/src/mempool.rs
* restores `tip_rejected_exact` type
* updates affected tests
* Documents the new argument in `Storage::insert()`, updates outdated comment
* Update zebrad/src/components/mempool/storage/verified_set.rs
* fixes potential issue with calling buffered mempool's poll_ready() method without calling it.
* Avoids removing dependent transactions of transactions that have been mined onto the best chain.
* Updates `spent_utxos()` method documentation
* Avoids sorting getblocktemplate transactions in non-test compilations
* documents PendingOutputs struct
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
* cargo fmt
* Applies suggestions from code review
Avoids unnecessarily rejecting dependent transactions of randomly evicted mempool transactions.
Updates `TransactionDependencies::remove_all()` to omit provided transaction id from the list of removed transaction ids.
* Applies suggestions from code review.
* Adds minor comments
* Update zebrad/src/components/mempool/storage/verified_set.rs
Co-authored-by: Marek <mail@marek.onl>
* Remove an outdated comment (#9013)
---------
Co-authored-by: Marek <mail@marek.onl>
In some cases Zebra logs might not output the database version, and thus we should avoid creating a disk without a version.
Before this change, a disk was created without a db version number, just indicating a `-v-`, that caused other tests to fail as an actual version was not found in their regexes.
Previously, most of our deployed instances needed to sync the whole blockchain from genesis, but after implementing the mounting of cached states for the release instances, this is no longer required.
Main changes:
- Reduce the boot disk size for CD images to 10GB (in CI Zebra might need to rebuild based on test flags, requiring more disk space)
- Use `pd-standard` instead of `pd-ssd` for the boot disk
- Use `pd-balanced` instead of `pd-ssd` for the mounted disk (where most of the reads and writes happens)
- Change our `GCP_SMALL_MACHINE` from `c2-standard-4` (vCPUs: 4, RAM: 16 GiB) to `c2d-standard-2` (vCPUs: 2, RAM: 8 GiB)
- Keep long running tests `is_long_test` with `GCP_LARGE_MACHINE` (`c2d-standard-16`) and other with the new `GCP_SMALL_MACHINE` configuration (`c2d-standard-2`)
* book: add section about private testnet testing
* Apply suggestions from code review
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Arya <aryasolhi@gmail.com>
---------
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
Co-authored-by: Arya <aryasolhi@gmail.com>
* Remove temporary dependency patches
We need to enable the `legacy-api` feature of `incrementalmerkletree` to
be able to serialize note commitment trees using an old serialization
format for the `z_gettreestate` RPC.
* Use spaces instead of a tab
* Bump ECC deps to match `zcashd`
* update documentation related auth cookie method for rpcs in the zebra book
* Update book/src/user/lightwalletd.md
* update rpc auth statement
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
Arya
Alfredo Garcia
Marek
Gustavo Valverde
Conrado Gouvea
Cypher Pepe
pinglanlu
dependabot[bot]
idky137