* fix(ci): correctly use lowered network caps
In the Test workflow we were using a different approach than the one being used in the Full sync test.
Also, in the Full sync test the variable was LOWER_NET_NAME, but NETWORK was being used in the disk name, with caps.
* imp(ci): get state version from local constants.rs
* imp(ci): use the same get name approach
* fix(ci): use the correct name for state version variable
* imp(ci)!: use different disk names for cached states
Disk states synced to canopy and synced to the chain tip should have different names to reference correctly on actual and coming tests the needed disk.
* imp(ci): test-stateful-sync no longer depends on regenerate-stateful-disks
* Apply suggestions from code review
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* fix(ci): use a better name for network string conversion
* Revert "Apply suggestions from code review"
This reverts commit cbbfaf4e9c.
* fix: do not get log information if sync was skipped
* fix(ci): do not lower the variable name
* fix(ci): use the same lowering case for network everywhere
* test: more .dockerignore conditions
* fix: use the right approach to lower caps
* remove extra .dockerignore
* trigger a change for stateful disk regeneration
* imp(ci): use `checkpoint` as the disk reference
* revert wrong delete
* fix(ci): add INSTANCE_ID and correct logging message
* imp(ci): add `v` prefix to state version number
* fix(ci): remove typo from logging message to get the height
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* style(ci): comply with https://json.schemastore.org/github-workflow.json
Some substituions were harder to make as files were not standardized
* fix(mergify): use correct name for macos
* style(actions): revert to single quotes
* style: lint dependabot and mergify conf files
* style: remove conditions with missing context
* imp(lint): automate GH Actions linting
* fix(lint): some actions need to be triggered by PR event
* fix(lint): consider all workflow YAMLs
* Use the same paths in the patch file
* revert: keep condition as is
* add TODO
* fix: add missing checkpoint_sync input
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add a rpc queue
* Implement the rpc queue
* Add rpc queue tests
* Remove mutex, use broadcast channel
* Have order and limit in the queue
* fix multiple transactions channel
* Use a network argument
* Use chain tip to calculate block spacing
* Add extra time
* Finalize the state check test
* Add a retry test
* Fix description
* fix some docs
* add additional empty check to `Runner::run`
* remove non used method
* ignore some errors
* fix some docs
* add a panic checker to the queue
* add missing file changes for panic checker
* skip checks and retries if height has not changed
* change constants
* reduce the number of queue test cases
* remove suggestion
* change best tip check
* fix(rpc): Check for panics in the transaction queue (#4046)
* Check for panics in the RPC transaction queue
* Add missing pin! and abort in the start task
* Check for transaction queue panics in tests
* Fixup a new RPC test from the main branch
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Add Utxo constructors from output locations
* Store transparent outputs rather than Utxo structs
* Update raw data snapshots
* Increment the state version
* Change OutputLocation to contain a TransactionLocation
* Change OutputLocation reads from the database
* Update some doc comments
* Update some TODOs
* Change deleting spent UTXOs and updating spent balances
* Change adding new UTXOs and adding their values to balances
* Disable dead code warnings
* Update snapshot test code
* Update round-trip tests for OutputLocations
* Update snapshot test data
* Increment the database format version
* Remove a redundant try_into()
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* Refactor redundant code
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* ci: attempt at fixing 'Regenerate stateful disks'
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* fix(ci): validate exit code
* fix(ci): use a single step
* fix: run from gcp command
* fix: remove variable substitution
* fix: escaping characters
* fix: handle bash issues with variable expansion
* fix: seggregate execution into multiple steps
* fix: the command is a string
* fix: move docker ignore to use caching
* revert panic
* fix: add panic and fix exit code command
* revert: remove panic
* fix: apply the exit code to all gcp tests
* imp(ci): use single line exit
* Clean up the CODEOWNERS file
We moved the dockerignore file into the docker directory,
so it doesn't need a separate entry any more.
Co-authored-by: teor <teor@riseup.net>
* Add an empty balance_by_transparent_addr column family
* Add an AddressBalanceLocation type for balance_by_transparent_addr
* Add serialization for balance_by_transparent_addr types
* Add round-trip tests for the new serialized types
* Add missing round-trip and serialized equality tests
* Add a network field to DiskWriteBatch
* Refactor confusing all_utxos_spent_by_block argument
It was actually just the UTXOs from the state spent by the block,
excluding the UTXOs created and spent within the block.
But now we need it to contain all the spent outputs,
including the ones created by the block.
* Read and update address balances in the finalized state
* Update raw data snapshots for transparent address balances
* Add test-only deserialization for transparent addresses
* Add high-level snapshot test code for address balances
* Add high-level snapshots for address balances
* Increment the state version after NU5 testnet 2 rollback
* Implement disk serialization for block headers and transactions
* Re-order column family initialization to match the design
* Add new empty transaction column families
* Split writing block header and transaction data
* Re-order column families for consistency
* Update write snapshots for transaction split
* Use split block and transaction data when reading
* Update snapshots to include genesis transaction hash location
* Filter all prefix iterators to make sure they return the correct values
* Test that the new transaction indexes are consistent
* Add some cleanup TODOs
* Increment the database format to version 15
* Remove unused fisk format impls for Block
* Add a missing prefix extractor for transaction locations
* Make the database generic over the thread mode
* Replace prefix iteration with iteration from a key, and a filter
Prefix iteration caused database hangs.
* Manually iterate through transaction locations to re-create blocks
Also:
- re-write disk read API to avoid iterator hangs
- move disk read API to ReadDisk
- re-write impl rocksdb::AsColumnFamilyRef to a where clause, for consistency
* Update the database version so it's larger than the NU5 testnet 2 version
* Create disk image after a successful full sync test
* Extract full sync height and name zebrad cached state with it
* Read 500 lines to extract sync height
* Restrict log query to just the container output and fix regex syntax for ubuntu
* Explicitly search logs in descending time
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Update changelog for v1.0.0-beta.7
* Increment all crate versions
* Remove redundant release test that is now covered by CI
* Remove completed NU5 README check task from the release template
* Add Merge Freeze tool to the release checklist
* Simplify release checklist by removing unused steps
Only run the state rebuild job if the database format version has (likely) changed.
If we have accidentally changed the format, but not changed the version,
we want to run with the old cached state, so this job fails.
If we change the state path without changing the version,
this job will take a few hours, because it will do a full rebuild.
When a PR is created and an image is built in a branch, the cache is also pushed to a `buildcache` tag.
As all PRs are using the same tag, sometimes the cache for a specific branch gets invalidated and makes it take longer on further pushes.
This fix might make the first commit take longer, but further ones will be faster if no changes in the code are applied.
* Return error code -8 for missing blocks, like zcashd does
* Test that getblock returns error code -8 for missing blocks
* Make RegexSet failures easier to read in logs
* Update lightwalletd acceptance tests for log message changes
* Add extra failure strings for lightwalletd logs
* Improve link formatting
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
* Make proptest-impl features consistently depend on each other
* Use a non-test method for consensus checks
* Make proptest dependencies the same between crates
* Repoint zebra image links to our new zfnd.org site for now
* Remove images/
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Re-order column families in design in dependency order
* Minor RFC design tweaks and fixes
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Update test.patch.yml with lightwalletd job
* Remove a workflow condition that will always be false
In general, patch workflows need the
opposite conditions to the original workflow.
But in this case, we know the result of the
condition will always be true, so we can just delete it.
Co-authored-by: teor <teor@riseup.net>
- use info level, there is nothing the user needs to do,
particularly for a single error
- explain that the errors are temporary
- hide backtraces, because they look like crashes
* Implement `getblockchaininfo` RPC method
* add a test for `get_blockchain_info`
* fix tohex/fromhex
* move comment
* Update lightwalletd acceptance test for getblockchaininfo RPC (#3914)
* change(rpc): Return getblockchaininfo network upgrades in height order (#3915)
* Update lightwalletd acceptance test for getblockchaininfo RPC
* Update some doc comments for network upgrades
* List network upgrades in order in the getblockchaininfo RPC
Also:
- Use a constant for the "missing consensus branch ID" RPC value
- Simplify fetching consensus branch IDs
- Make RPC type derives consistent
- Update RPC type documentation
* Make RPC type derives consistent
* Fix a confusing test comment
* get hashand height at the same time
* fix estimated_height
* fix lint
* add extra check
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix typo
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* split test
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
* fix(rpc): ignore an expected error in the RPC acceptance tests (#3961)
* Add ignored regexes to test command failure regex methods
* Ignore empty chain error in getblockchaininfo
We expect this error when zebrad starts up with an empty state.
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Make command test matching code accept generic regexes
And add generic conversions to regexes.
* Document test command structs
* Support matching multiple regexes internally in the test command
* Make it easier to call the generic regex methods
* Add a missing API usage comment
* Fix a potential hang in test child error reports
* Revert Option<Child> process handling
* Revert "Revert Option<Child> process handling"
This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.
* Add a set of failure regexes to test command output
* Allow debug-printing TestChild again
* When the child is dropped, check any remaining output
* Document a wait_with_output edge case
* Improve failure regex panic output
* Improve builder ergonomics
* Add internal tests for failure regex panics
It would be easy to disable these panics, and never realise.
* Add some module structure TODOs
* Stop panicking if the child process has already been taken
* Add test APIs for consuming child output lines
* Fix a hang on child process drop
* Handle output being already taken in wait_with_output
And document some edge cases we don't handle yet
* Use bash's read command in the TestChild stderr test
And check the actual command we're using to see if it errors.
* Pretty print full failure regex list
* Add the test child command line to the failure regex logs
* Simplify the sync_until test code
* Check logs for failure messages in lightwalletd integration tests
* Add more TODOs and improve comments
* Allow missing branch ID 00000000 provided by zcashd
* Revert "Revert Option<Child> process handling"
This reverts commit 2af30086858d104dcb0ec87383996c36bcaa7371.
* Add a set of failure regexes to test command output
* Allow debug-printing TestChild again
* When the child is dropped, check any remaining output
* Document a wait_with_output edge case
* Improve failure regex panic output
* Improve builder ergonomics
* Add internal tests for failure regex panics
It would be easy to disable these panics, and never realise.
* Add some module structure TODOs
* Stop panicking if the child process has already been taken
* Add test APIs for consuming child output lines
* Fix a hang on child process drop
* Handle output being already taken in wait_with_output
And document some edge cases we don't handle yet
* Use bash's read command in the TestChild stderr test
And check the actual command we're using to see if it errors.
* Pretty print full failure regex list
* Add the test child command line to the failure regex logs
* Make command test matching code accept generic regexes
And add generic conversions to regexes.
* Document test command structs
* Support matching multiple regexes internally in the test command
* Make it easier to call the generic regex methods
* Add a missing API usage comment
* Fix a potential hang in test child error reports
* Revert Option<Child> process handling
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Gustavo Valverde
Alfredo Garcia
teor
dependabot[bot]
Deirdre Connolly
Conrado Gouvea
Dimitris Apostolou
tokidoki