* added functions for fetching block vectors
* inserted new network methods for vector fetching into zebra-chain
* changed tag back to test/feature=branch
* changed tag back to test/feature=branch
* changed tag back to test/feature=branch
* changed feature tag to proptest-impl, started implementing in zebra-consensus tests
* adding methods to zebra-consensus, lifetime error in src/transaction/tests.rs, needs refactoring
* finished adding methods to zebra-consensus
* finished adding new methods to zebrad
* added new methods to zebra-rpc and zebra-scan
* finished removing statements matching on Network from tests
* updated new error message
* changed get_block_bytes() and get_block_sapling_roots_bytes to return option and removed serialization error types as per requested changes in PR review
* removed match statements from zebra_chain::transaction::arbitrary::test_transactions() and new zebra-grpc tests
* moved zebra-chain::test_utils to zebra-chain::test
* removed get_ prefix from getter methods
* renamed zebra-chain::test to zebra-chain::tests
* renamed zebra-chain::test to zebra-chain::tests
* fixed clippy warnings
* changed block_map to return clone
* add `sapling_subtree_for_tip` and `orchard_subtree_for_tip` methods to `ZebraDb`
* add methods for non finalized state, move functions
* call `zs_last_key_value` the right way
* fix and simplify `*_subtree_for_tip` methods
Co-authored-by: Arya <aryasolhi@gmail.com>
* apply filter
* rename all tree and subtree methods that use tip
* rename tip tree and subtree methods in non finalized chain
* apply simplify suggestions
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* Pass ZebraDB to batch preparation
* Dedup the insertion of Sapling trees into database
* Dedup the insertion of Orchard trees into database
* Update snapshots
* Rename batch preparation of trees
* Simplify the naming of note commitment trees
* Correctly retrieve Sapling trees from fin state
* Correctly retrieve Orchard trees from fin state
* Simplify the naming of methods for Sprout trees
* Simplify the naming of methods for Sapling trees
* Simplify the naming of methods for Orchard trees
* Reduce disk reads by caching trees. (#7276)
* Bump the state minor version
* Reset the state patch version
* Simplify the preparation of genesis trees
* Store the roots of the trees of the genesis block
* Add the genesis roots to snapshots
* fix(test): Don't include shielded data in genesis blocks (#7302)
* fix(state): Fix marking format upgrades (#7304)
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Add methods for loading entire column families from the database
* Add a method that loads all the sprout trees from the database
* Add snapshot tests for sprout note commitment trees
* Add round-trip proptests for tree root database serialization
* Add a manual sprout note commitment tree database serialization snapshot test
* Add tests for 1,2,4,8 note commitments in a tree
* Remove redundant "rand" package rename in dependencies
* Randomly cache roots rather than only caching even roots
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* ZIPs were updated to remove ambiguity, this was tracked in #1267.
* #2105 was fixed by #3039 and #2379 was closed by #3069
* #2230 was a duplicate of #2231 which was closed by #2511
* #3235 was obsoleted by #2156 which was fixed by #3505
* #1850 was fixed by #2944, #1851 was fixed by #2961 and #2902 was fixed by #2969
* We migrated to Rust 2021 edition in Jan 2022 with #3332
* #1631 was closed as not needed
* #338 was fixed by #3040 and #1162 was fixed by #3067
* #2079 was fixed by #2445
* #4794 was fixed by #6122
* #1678 stopped being an issue
* #3151 was fixed by #3934
* #3204 was closed as not needed
* #1213 was fixed by #4586
* #1774 was closed as not needed
* #4633 was closed as not needed
* Clarify behaviour of difficulty spacing
Co-authored-by: teor <teor@riseup.net>
* Update comment to reflect implemented behaviour
Co-authored-by: teor <teor@riseup.net>
* Update comment to reflect implemented behaviour when retrying block downloads
Co-authored-by: teor <teor@riseup.net>
* Update `TODO` to remove closed issue and clarify when we might want to fix
Co-authored-by: teor <teor@riseup.net>
* Update `TODO` to remove closed issue and clarify what we might want to change in future
Co-authored-by: teor <teor@riseup.net>
* Clarify benefits of how we do block verification
Co-authored-by: teor <teor@riseup.net>
* Fix rustfmt errors
---------
Co-authored-by: teor <teor@riseup.net>
* add initial ES support
* hide elasticsearch behind feture, add config
* make the builds and clippy happy
* move ES code to a function
* change database name
* fix database name and panic
* increase close to tip interval
* update deny for elastic
* remove a block clone
* try to fix builds
* fix builds 2
* refactor some imports and unwraps
* typo
* change argument of elasticsearch function
* move elastic call to the end of `commit_finalized_direct`
* get height from block
* cargo upgrade --workspace rocksdb
* Add a deny.toml exception for bindgen 0.59
* Move `valid()` and `status()` methods to raw iterators
* Update some outdated comments
* Panic on iterator failures
We might want to change this if there are any common failure modes.
* allow(clippy::unwrap_in_result) in some methods
* Split disk reads from CPU-heavy Sprout interstitial tree cryptography
* Improve anchor validation debugging and error messages
* Work around a test data bug, and save some CPU
* Remove redundant checks for empty shielded data
* Skip generating unused interstitial treestates
* Do disk fetches and quick checks, then CPU-heavy cryptography
* Wrap HistoryTree in an Arc in the state
* Run CPU-intensive chain validation and updates in parallel rayon threads
* Refactor to prepare for parallel tree root calculations
* Run finalized state note commitment tree root updates in parallel rayon threads
* Update finalized state note commitment trees using parallel rayon threads
* Fix a comment typo and add a TODO
* Split sprout treestate fetch into its own function
* Move parallel note commitment trees to zebra-chain
* Re-calculate the tree roots in the same parallel batches
* Do non-finalized note commitment tree updates in parallel threads
* Update comments about note commitment tree rebuilds
* Do post-fork tree updates in parallel threads
* Add a TODO for parallel tree updates in tests
* Fix broken intra-doc links
* Clarify documentation for sprout treestates
* Sort Cargo.toml dependencies
* Wrap Sprout note commitment trees into `Arc`
* Remove a redundant comment
* Rephrase a comment about chain forking
* Remove a redundant comment
The comment is not valid because Zebra uses `bridgetree::Frontier`s from
the `incrementalmerkletree` crate to represent its note commitment
trees. This `struct` does not support popping elements from the tree.
* Wrap Sapling commitment trees into `Arc`
* Remove unnecessary `as_ref`s
* Wrap Orchard commitment trees into `Arc`
* Fix the syntax of links in comments
* Fix a mistake in the docs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Remove unnecessary angle brackets from a link
* Revert the changes for links that serve as references
* Revert "Revert the changes for links that serve as references"
This reverts commit 8b091aa9fa.
* Remove `<` `>` from links that serve as references
This reverts commit 046ef25620.
* Don't use `<` `>` in normal comments
* Don't use `<` `>` for normal comments
* Revert changes for comments starting with `//`
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Add a finalized state txids query
* Add an address transaction IDs query, without height filters
* Connect the address transaction ID query to the RPC
* Basic filtering of address transaction IDs by height range
* Add a network and range argument to the getaddresstxids test
* Test all block range combinations for mainnet
* Fix a file descriptor limit error
* Optimise seeking the first transaction for an address
The first transaction's location is part of the address location.
* Filter finalized address transaction IDs by height range
* Filter non-finalized address transaction IDs by the height range
* Fix up snapshot tests for the new height range API
* Add test-only serialization, and make existing serialization test-only
* Make AddressLocations clearer in the API
* Add UnspentOutputAddressLocation
* Add the AddressLocation to the UTXO database value
* Update the snapshot test code for UnspentOutputAddressLocation
* Update the raw data snapshots
* Update the high-level data snapshots
* Increment the database version
* Make serialization clearer
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Fix code formatting
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add an empty utxo_by_transparent_addr_loc column family
* Update snapshot data for the new column family
* Add an AddressUnspentOutputs type
* Add round-trip tests for AddressUnspentOutputs
* Move address balances into their own method
* Simplify updating address balances
* Fix utxo_by_out_loc column family name
* Implement reads and writes of address UTXOs
* Update raw data snapshots
* Update the snapshot tests for high-level address UTXOs
* Assert rather than taking empty address snapshots for genesis
* Update high-level address UTXO snapshot data, and delete empty snapshots
* Increment the database version
* Use typed values for all ReadDisk methods
* Implement test-only serialization for transparent::Address
* Implement FromDisk for ()
* Store AddressUnspentOutput as the column family key
* Update round-trip serialization tests for AddressUnspentOutput
* Update snapshot test code, and add a UTXO data snapshot
* Update existing snapshot data
* Add new UTXO snapshot data
* Update column family name
```sh
fastmod utxo_by_transparent_addr_loc utxo_loc_by_transparent_addr_loc zebra*
```
* cargo fmt --all
* cargo insta test --review --delete-unreferenced-snapshots
* Explain why it is ok to use invalid database iterator indexes
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add explanations of UTXO database updates
* Simplify an assertion
* Remove UnspentOutputAddressLocation and just store transparent::Output
* Update snapshot test data
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Derive Hash for transparent address index types
* Expose some types used by transparent address indexes
* Add an empty transparent transfers type for transparent address indexes
* Update TransparentTransfers with created UTXOs
* Add spent transparent outputs to ContextuallyValidBlock
* Update TransparentTransfers with spent transparent outputs
* Ignore missing spent outputs, so that tests pass
* Remove empty TransparentTransfers after a spend revert
* Update TransparentTransfers with creating and spending transaction IDs
* Ignore duplicate created UTXOs, so that tests pass
* Add some TODO comments
* Remove accidental doctest formatting
* Add address transfers index accessor methods
* Use TransactionLocation in the non-finalized state
* Apply more address index assertions to production code
* Refactor deeply nested code and apply more assertions
* Return UTXOs in chain order
* Return transaction hashes in chain order
* Stop indexing each transparent output multiple times
* Run some more asserts during tests
* Tidy TODO comments
* Fix an incorrect assert condition
* Use OrderedUtxos so that spent UTXOs can be stored in chain order
* Update tests to use OrderedUtxos
* Update the index API for the getaddressutxos query
* Remove redundant arguments in tests
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Change OutputLocation to contain a TransactionLocation
* Change OutputLocation reads from the database
* Update some doc comments
* Update some TODOs
* Change deleting spent UTXOs and updating spent balances
* Change adding new UTXOs and adding their values to balances
* Disable dead code warnings
* Update snapshot test code
* Update round-trip tests for OutputLocations
* Update snapshot test data
* Increment the database format version
* Remove a redundant try_into()
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Refactor redundant code
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* ci: attempt at fixing 'Regenerate stateful disks'
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add an empty balance_by_transparent_addr column family
* Add an AddressBalanceLocation type for balance_by_transparent_addr
* Add serialization for balance_by_transparent_addr types
* Add round-trip tests for the new serialized types
* Add missing round-trip and serialized equality tests
* Add a network field to DiskWriteBatch
* Refactor confusing all_utxos_spent_by_block argument
It was actually just the UTXOs from the state spent by the block,
excluding the UTXOs created and spent within the block.
But now we need it to contain all the spent outputs,
including the ones created by the block.
* Read and update address balances in the finalized state
* Update raw data snapshots for transparent address balances
* Add test-only deserialization for transparent addresses
* Add high-level snapshot test code for address balances
* Add high-level snapshots for address balances
* Increment the state version after NU5 testnet 2 rollback
* Implement disk serialization for block headers and transactions
* Re-order column family initialization to match the design
* Add new empty transaction column families
* Split writing block header and transaction data
* Re-order column families for consistency
* Update write snapshots for transaction split
* Use split block and transaction data when reading
* Update snapshots to include genesis transaction hash location
* Filter all prefix iterators to make sure they return the correct values
* Test that the new transaction indexes are consistent
* Add some cleanup TODOs
* Increment the database format to version 15
* Remove unused fisk format impls for Block
* Add a missing prefix extractor for transaction locations
* Make the database generic over the thread mode
* Replace prefix iteration with iteration from a key, and a filter
Prefix iteration caused database hangs.
* Manually iterate through transaction locations to re-create blocks
Also:
- re-write disk read API to avoid iterator hangs
- move disk read API to ReadDisk
- re-write impl rocksdb::AsColumnFamilyRef to a where clause, for consistency
* Update the database version so it's larger than the NU5 testnet 2 version
* Add round-trip test for OutputLocation disk serialization
* Make the transaction snapshot tests more accurate
Previously, we were showing the genesis transaction hash at height 0, index 0.
But its hash is actually not stored by location in the database,
because the genesis transaction is skipped due to a consensus rule.
* Update the transaction snapshot data
* Add history tree snapshot tests
At the current test heights, the history trees are all empty.
* Add the history tree snapshot data
* Update comments
* Simplify snapshot test code
* Make some serde::Serialize impls test-only, so we can hex-encode them
This should also speed up release compilation a bit.
* Add snapshot test code for UTXOs
* Add snapshot data for UTXOs
* Query Sapling & Orchard trees by height in the finalized state
* Add Sapling & Orchard trees to the non-finalized state
* Add a TODO about concurrent read-only access to Sprout tree
Co-authored-by: teor <teor@riseup.net>
* Update the database format version
* Keep only the most recent Sprout tree in the database
* Check that the database returns empty trees for the genesis block
* Assert that the database returns the highest trees
* Document how to update insta snapshots
* Add note commitment tree insta snapshot tests
* Add comments about cached tree roots in snapshots
* Add snapshot data for sapling and orchard trees
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* doc(db): fix some comments
* refactor(db): split disk serialization types into their own module
* refactor(db): split the disk format into modules
* doc(db/test): explain the RON serialization format
* test(db): snapshot finalized tips
* test(db): snapshot block heights and hashes
* test(db): snapshot block data
* test(db): snapshot transaction hashes and data
* test(db): assert that blocks and transactions are in chain order
* clippy(db/test): cargo clippy --fix
* clippy(db/test): replace &Vec with &[_]
idky137
Alfredo Garcia
Marek
teor
Pili Guerra
Conrado Gouvea