* feat(ci): run cached state rebuilds in main branch
* fix(ci): allow the PR/branch name in the disk name
Move the hight information to the disk description, to reduce the name length
* fix(ci): add missing SHA
* fix(ci): regenerate chekpoint cached state on main
This will automatically regenerate the disk when a merge is completed on main
* tmp(ci): do not duplicate sync test at/after merge
This temporarily ensure the test just runs in the main branch, ensuring we can track it easier
* implement `getaddresstxids` rpc method with dummy empty response
* use already public function
* fix some docs
* pass a list of addresses to the state request
* sync range errors with zcashd
* refactor a loop
* fix grammar
* fix tests
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
* Add test-only serialization, and make existing serialization test-only
* Make AddressLocations clearer in the API
* Add UnspentOutputAddressLocation
* Add the AddressLocation to the UTXO database value
* Update the snapshot test code for UnspentOutputAddressLocation
* Update the raw data snapshots
* Update the high-level data snapshots
* Increment the database version
* Make serialization clearer
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Fix code formatting
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Add an empty utxo_by_transparent_addr_loc column family
* Update snapshot data for the new column family
* Add an AddressUnspentOutputs type
* Add round-trip tests for AddressUnspentOutputs
* Move address balances into their own method
* Simplify updating address balances
* Fix utxo_by_out_loc column family name
* Implement reads and writes of address UTXOs
* Update raw data snapshots
* Update the snapshot tests for high-level address UTXOs
* Assert rather than taking empty address snapshots for genesis
* Update high-level address UTXO snapshot data, and delete empty snapshots
* Increment the database version
* Use typed values for all ReadDisk methods
* Implement test-only serialization for transparent::Address
* Implement FromDisk for ()
* Store AddressUnspentOutput as the column family key
* Update round-trip serialization tests for AddressUnspentOutput
* Update snapshot test code, and add a UTXO data snapshot
* Update existing snapshot data
* Add new UTXO snapshot data
* Update column family name
```sh
fastmod utxo_by_transparent_addr_loc utxo_loc_by_transparent_addr_loc zebra*
```
* cargo fmt --all
* cargo insta test --review --delete-unreferenced-snapshots
* Explain why it is ok to use invalid database iterator indexes
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Add explanations of UTXO database updates
* Simplify an assertion
* Remove UnspentOutputAddressLocation and just store transparent::Output
* Update snapshot test data
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Derive Hash for transparent address index types
* Expose some types used by transparent address indexes
* Add an empty transparent transfers type for transparent address indexes
* Update TransparentTransfers with created UTXOs
* Add spent transparent outputs to ContextuallyValidBlock
* Update TransparentTransfers with spent transparent outputs
* Ignore missing spent outputs, so that tests pass
* Remove empty TransparentTransfers after a spend revert
* Update TransparentTransfers with creating and spending transaction IDs
* Ignore duplicate created UTXOs, so that tests pass
* Add some TODO comments
* Remove accidental doctest formatting
* Add address transfers index accessor methods
* Use TransactionLocation in the non-finalized state
* Apply more address index assertions to production code
* Refactor deeply nested code and apply more assertions
* Return UTXOs in chain order
* Return transaction hashes in chain order
* Stop indexing each transparent output multiple times
* Run some more asserts during tests
* Tidy TODO comments
* Fix an incorrect assert condition
* Use OrderedUtxos so that spent UTXOs can be stored in chain order
* Update tests to use OrderedUtxos
* Update the index API for the getaddressutxos query
* Remove redundant arguments in tests
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix(ci): correctly use lowered network caps
In the Test workflow we were using a different approach than the one being used in the Full sync test.
Also, in the Full sync test the variable was LOWER_NET_NAME, but NETWORK was being used in the disk name, with caps.
* imp(ci): get state version from local constants.rs
* imp(ci): use the same get name approach
* fix(ci): use the correct name for state version variable
* imp(ci)!: use different disk names for cached states
Disk states synced to canopy and synced to the chain tip should have different names to reference correctly on actual and coming tests the needed disk.
* imp(ci): test-stateful-sync no longer depends on regenerate-stateful-disks
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* fix(ci): use a better name for network string conversion
* Revert "Apply suggestions from code review"
This reverts commit cbbfaf4e9c.
* fix: do not get log information if sync was skipped
* fix(ci): do not lower the variable name
* fix(ci): use the same lowering case for network everywhere
* test: more .dockerignore conditions
* fix: use the right approach to lower caps
* remove extra .dockerignore
* trigger a change for stateful disk regeneration
* imp(ci): use `checkpoint` as the disk reference
* revert wrong delete
* fix(ci): add INSTANCE_ID and correct logging message
* imp(ci): add `v` prefix to state version number
* fix(ci): remove typo from logging message to get the height
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* style(ci): comply with https://json.schemastore.org/github-workflow.json
Some substituions were harder to make as files were not standardized
* fix(mergify): use correct name for macos
* style(actions): revert to single quotes
* style: lint dependabot and mergify conf files
* style: remove conditions with missing context
* imp(lint): automate GH Actions linting
* fix(lint): some actions need to be triggered by PR event
* fix(lint): consider all workflow YAMLs
* Use the same paths in the patch file
* revert: keep condition as is
* add TODO
* fix: add missing checkpoint_sync input
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a rpc queue
* Implement the rpc queue
* Add rpc queue tests
* Remove mutex, use broadcast channel
* Have order and limit in the queue
* fix multiple transactions channel
* Use a network argument
* Use chain tip to calculate block spacing
* Add extra time
* Finalize the state check test
* Add a retry test
* Fix description
* fix some docs
* add additional empty check to `Runner::run`
* remove non used method
* ignore some errors
* fix some docs
* add a panic checker to the queue
* add missing file changes for panic checker
* skip checks and retries if height has not changed
* change constants
* reduce the number of queue test cases
* remove suggestion
* change best tip check
* fix(rpc): Check for panics in the transaction queue (#4046)
* Check for panics in the RPC transaction queue
* Add missing pin! and abort in the start task
* Check for transaction queue panics in tests
* Fixup a new RPC test from the main branch
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add Utxo constructors from output locations
* Store transparent outputs rather than Utxo structs
* Update raw data snapshots
* Increment the state version
* Change OutputLocation to contain a TransactionLocation
* Change OutputLocation reads from the database
* Update some doc comments
* Update some TODOs
* Change deleting spent UTXOs and updating spent balances
* Change adding new UTXOs and adding their values to balances
* Disable dead code warnings
* Update snapshot test code
* Update round-trip tests for OutputLocations
* Update snapshot test data
* Increment the database format version
* Remove a redundant try_into()
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Refactor redundant code
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* ci: attempt at fixing 'Regenerate stateful disks'
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* fix(ci): validate exit code
* fix(ci): use a single step
* fix: run from gcp command
* fix: remove variable substitution
* fix: escaping characters
* fix: handle bash issues with variable expansion
* fix: seggregate execution into multiple steps
* fix: the command is a string
* fix: move docker ignore to use caching
* revert panic
* fix: add panic and fix exit code command
* revert: remove panic
* fix: apply the exit code to all gcp tests
* imp(ci): use single line exit
* Clean up the CODEOWNERS file
We moved the dockerignore file into the docker directory,
so it doesn't need a separate entry any more.
Co-authored-by: teor <teor@riseup.net>
* Add an empty balance_by_transparent_addr column family
* Add an AddressBalanceLocation type for balance_by_transparent_addr
* Add serialization for balance_by_transparent_addr types
* Add round-trip tests for the new serialized types
* Add missing round-trip and serialized equality tests
* Add a network field to DiskWriteBatch
* Refactor confusing all_utxos_spent_by_block argument
It was actually just the UTXOs from the state spent by the block,
excluding the UTXOs created and spent within the block.
But now we need it to contain all the spent outputs,
including the ones created by the block.
* Read and update address balances in the finalized state
* Update raw data snapshots for transparent address balances
* Add test-only deserialization for transparent addresses
* Add high-level snapshot test code for address balances
* Add high-level snapshots for address balances
* Increment the state version after NU5 testnet 2 rollback
* Implement disk serialization for block headers and transactions
* Re-order column family initialization to match the design
* Add new empty transaction column families
* Split writing block header and transaction data
* Re-order column families for consistency
* Update write snapshots for transaction split
* Use split block and transaction data when reading
* Update snapshots to include genesis transaction hash location
* Filter all prefix iterators to make sure they return the correct values
* Test that the new transaction indexes are consistent
* Add some cleanup TODOs
* Increment the database format to version 15
* Remove unused fisk format impls for Block
* Add a missing prefix extractor for transaction locations
* Make the database generic over the thread mode
* Replace prefix iteration with iteration from a key, and a filter
Prefix iteration caused database hangs.
* Manually iterate through transaction locations to re-create blocks
Also:
- re-write disk read API to avoid iterator hangs
- move disk read API to ReadDisk
- re-write impl rocksdb::AsColumnFamilyRef to a where clause, for consistency
* Update the database version so it's larger than the NU5 testnet 2 version
* Create disk image after a successful full sync test
* Extract full sync height and name zebrad cached state with it
* Read 500 lines to extract sync height
* Restrict log query to just the container output and fix regex syntax for ubuntu
* Explicitly search logs in descending time
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Update changelog for v1.0.0-beta.7
* Increment all crate versions
* Remove redundant release test that is now covered by CI
* Remove completed NU5 README check task from the release template
* Add Merge Freeze tool to the release checklist
* Simplify release checklist by removing unused steps
Only run the state rebuild job if the database format version has (likely) changed.
If we have accidentally changed the format, but not changed the version,
we want to run with the old cached state, so this job fails.
If we change the state path without changing the version,
this job will take a few hours, because it will do a full rebuild.
When a PR is created and an image is built in a branch, the cache is also pushed to a `buildcache` tag.
As all PRs are using the same tag, sometimes the cache for a specific branch gets invalidated and makes it take longer on further pushes.
This fix might make the first commit take longer, but further ones will be faster if no changes in the code are applied.
* Return error code -8 for missing blocks, like zcashd does
* Test that getblock returns error code -8 for missing blocks
* Make RegexSet failures easier to read in logs
* Update lightwalletd acceptance tests for log message changes
* Add extra failure strings for lightwalletd logs
* Improve link formatting
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Make proptest-impl features consistently depend on each other
* Use a non-test method for consensus checks
* Make proptest dependencies the same between crates
* Repoint zebra image links to our new zfnd.org site for now
* Remove images/
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Re-order column families in design in dependency order
* Minor RFC design tweaks and fixes
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Update test.patch.yml with lightwalletd job
* Remove a workflow condition that will always be false
In general, patch workflows need the
opposite conditions to the original workflow.
But in this case, we know the result of the
condition will always be true, so we can just delete it.
Co-authored-by: teor <teor@riseup.net>
Gustavo Valverde
Conrado Gouvea
Alfredo Garcia
teor
dependabot[bot]
Deirdre Connolly
Dimitris Apostolou