* fix: use `exit-nopipe` with consistent `shell` usage
Temporarily disabled the `set -e` option around the docker logs command to handle the broken pipe error gracefully.
Handle more complex scenarios in our `Result of ${{ inputs.test_id }} test` job
* fix: Use single quotes for the outer command
* fix: use same approach for CD
* test: check launch failure logs
* fix: revert CD changes
* fix: do not try to increase the disk size and wait mounting
* fix: increase GB a bit more
* fix: do not fail on pipe failure
* fix: use plain `tee /dev/stderr`
If this does not work try `(tee … || true)`
* fix: `tee` not stoping on cd config tests
* fix: match logic with GCP tests
* fix(cd): handle pipe and other errors correctly
* try `tee --output-error=exit-nopipe`
* fix: TRAP without pipefail
* test: pipefail with exit and trap
* fix: use a subshell
* fix(ci): wait for mounting and show system logs if fail
* fix(ci): GCP is not always mounting disks in the same order
* fix: use `grep` instead of `awk`
* fix: typo
* fix: use simpler `grep` command
* fix: do not sleep if not require
* chore: reduce diff
* add `sapling_subtree_for_tip` and `orchard_subtree_for_tip` methods to `ZebraDb`
* add methods for non finalized state, move functions
* call `zs_last_key_value` the right way
* fix and simplify `*_subtree_for_tip` methods
Co-authored-by: Arya <aryasolhi@gmail.com>
* apply filter
* rename all tree and subtree methods that use tip
* rename tip tree and subtree methods in non finalized chain
* apply simplify suggestions
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* add buld failures ticket generation to most important workflows
* add missing newline at the end of the file
* doc fixes
* add missing testnet build check
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: teor <teor@riseup.net>
* fix(ci): disk validation for docker volume mount
* Use a symlink for lightwalletd cached state rather than mounting the same volume twice
* Avoid "sdb seems to be busy" errors from docker by adding extra sleeps
* Add a missing backslash
* Remove symlink from workflow
* Symlink lightwalletd path in entrypoint.sh
* Retry on failure and check Docker logs
* End ssh shell lines with explicit terminators
* Delete Docker containers if Docker mount fails
* Revert symlink changes in entrypoint.sh
* Debug using lsof
* Use correct lsof commands
* Use correct syntax for lsof +D
* fix(ci): make multiple validations before mounting
Loop and checks for three conditions:
The device `/dev/sdb` exists.
No process is using the device `/dev/sdb`.
No process is using the Docker volume directory.
* fix: do not pre-mount docker volume
The Docker version available with the newer `cos-stable` OS (https://cloud.google.com/release-notes#cos-109-17800-0-45) allows to mount the image when running it.
Mounting it before makes the disk unavailable.
* fix: remove extra `;`
* fix: just confirm with `lsof` and show it's output
* chore: reduce diff
---------
Co-authored-by: teor <teor@riseup.net>
* Add a state full sync to the release-checklist.md
* Add state test and freeze to release.md
* Make testing requirement clearer
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Provide a user hint when the database format is invalid
* Split a path-based database version method
* Check the database format before Zebra shuts down
* Fix a typo in zebra-state/src/service/finalized_state/zebra_db.rs
Co-authored-by: Marek <mail@marek.onl>
* Fix a typo in upgrade.rs
---------
Co-authored-by: Marek <mail@marek.onl>
* Refactor format checks, stop running quick check before upgrade
* Speed up startup by always running the format checks in parallel
* Make detailed format checks cancellable on shutdown
* Prepare for periodic format checks
* Time upgrades and validity checks
* Run a database format check every 5 minutes
* Wait for a cancel signal rather than unconditionally sleeping
* Move check_max_on_disk_height() into the format checks
* Move spawn_format_change() into its own method
* Avoid manual handling of previous sapling trees by using iterator windows instead
* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead
* Simplify adding notes by using the exact number of remaining notes
* Simplify by skipping the first block, because it can't complete a subtree
* Re-use existing tree update code
* Apply the sapling changes to orchard subtree updates
* add a reverse database column family iterator function
* Make skipping the lowest tree independent of iteration order
* Move new subtree checks into the iterator, rename to end_height
* Split subtree calculation into a new method
* Split the calculate and write methods
* Quickly check the first subtree before running the full upgrade
* Do the quick checks every time Zebra runs, and refactor slow check error handling
* Do quick checks for orchard as well
* Make orchard tree upgrade match sapling upgrade code
* Upgrade subtrees in reverse height order
* Bump the database patch version so the upgrade runs again
* Reset previous subtree upgrade data before doing this one
* Add extra checks to subtree calculation to diagnose errors
* Use correct heights for subtrees completed at the end of a block
* Add even more checks to diagnose issues
* Instrument upgrade methods to improve diagnostics
* Prevent modification of re-used trees
* Debug with subtree positions as well
* Fix an off-by-one error with completed subtrees
* Fix typos and confusing comments
Co-authored-by: Marek <mail@marek.onl>
* Fix mistaken previous tree handling and end tree comments
* Remove unnecessary subtraction in remaining leaves calc
* Log heights when assertions fail
* Fix new subtree detection filter
* Move new subtree check into a method, cleanup unused code
* Remove redundant assertions
* Wait for subtree upgrade before testing RPCs
* Fix subtree search in quick check
* Temporarily upgrade subtrees in forward height order
* Clarify some comments
* Fix missing test imports
* Fix subtree logging
* Add a comment about a potential hang with future upgrades
* Fix zebrad var ownership
* Log more info when add_subtrees.rs fails
* cargo fmt --all
* Fix unrelated clippy::unnecessary_unwrap
* cargo clippy --fix --all-features --all-targets; cargo fmt --all
* Stop the quick check depending on tree de-duplication
* Refactor waiting for the upgrade into functions
* Wait for state upgrades whenever the cached state is updated
* Wait for the testnet upgrade in the right place
* Fix unused variable
* Fix a subtree detection bug and comments
* Remove an early reference to reverse direction
* Stop skipping subtrees completed at the end of blocks
* Actually fix new subtree code
* Upgrade subtrees in reverse height order
Reverts "Temporarily upgrade subtrees in forward height order"
This reverts commit a9558be214.
* Bump the database patch version to re-run the upgrade (for testing)
* Revert "Remove an early reference to reverse direction"
This reverts commit c206404377.
---------
Co-authored-by: Marek <mail@marek.onl>
* add fixed test vectors for `z_getsubtreesbyindex`
* change to snapshots
* add test to docker
* remove assert lines from snapshots
* add more tests
* change test description
* change test name
* run both tests together
* wait for state version update in test
* Run one test at a time to avoid state locking issues
---------
Co-authored-by: teor <teor@riseup.net>
Use `trap '' PIPE` as a command in the pipeline might exit early and we want the script to continue executing, as using `grep --max-count=1` might exit after finding the first match, causing a broken pipe error with `tee`.
This is being combined with `set -o pipefail` to have strict error handling and we want the script to fail if any command in the pipeline fails.
* Add instructions for changing crate owners
* Explain new owner invites
* Remove duplicate `cargo login` instructions from the release checklist
* Remove owners as well
* Missing log line
* imp(doc): add missing Docker variables and examples
* imp(doc): segment environment variables by role
* imp(doc): use code instead of strong styling for variables
* chore: review fixes
* fix(doc): use a better example for advanced usage
* Avoid manual handling of previous sapling trees by using iterator windows instead
* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead
* Simplify adding notes by using the exact number of remaining notes
* Simplify by skipping the first block, because it can't complete a subtree
* Re-use existing tree update code
* Apply the sapling changes to orchard subtree updates
* add a reverse database column family iterator function
* Make skipping the lowest tree independent of iteration order
* Move new subtree checks into the iterator, rename to end_height
* Split subtree calculation into a new method
* Split the calculate and write methods
* Quickly check the first subtree before running the full upgrade
* Do the quick checks every time Zebra runs, and refactor slow check error handling
* Do quick checks for orchard as well
* Make orchard tree upgrade match sapling upgrade code
* Upgrade subtrees in reverse height order
* Bump the database patch version so the upgrade runs again
* Reset previous subtree upgrade data before doing this one
* Add extra checks to subtree calculation to diagnose errors
* Use correct heights for subtrees completed at the end of a block
* Add even more checks to diagnose issues
* Instrument upgrade methods to improve diagnostics
* Prevent modification of re-used trees
* Debug with subtree positions as well
* Fix an off-by-one error with completed subtrees
* Fix typos and confusing comments
Co-authored-by: Marek <mail@marek.onl>
* Fix mistaken previous tree handling and end tree comments
* Remove unnecessary subtraction in remaining leaves calc
* Log heights when assertions fail
* Fix new subtree detection filter
* Move new subtree check into a method, cleanup unused code
* Remove redundant assertions
* Wait for subtree upgrade before testing RPCs
* Fix subtree search in quick check
* Temporarily upgrade subtrees in forward height order
* Clarify some comments
* Fix missing test imports
* Fix subtree logging
* Add a comment about a potential hang with future upgrades
* Fix zebrad var ownership
* Log more info when add_subtrees.rs fails
* cargo fmt --all
* Fix unrelated clippy::unnecessary_unwrap
* cargo clippy --fix --all-features --all-targets; cargo fmt --all
* Stop the quick check depending on tree de-duplication
* Refactor waiting for the upgrade into functions
* Wait for state upgrades whenever the cached state is updated
* Wait for the testnet upgrade in the right place
* Fix unused variable
* Fix a subtree detection bug and comments
* Remove an early reference to reverse direction
* Stop skipping subtrees completed at the end of blocks
* Actually fix new subtree code
---------
Co-authored-by: Marek <mail@marek.onl>
dependabot[bot]
Arya
Gustavo Valverde
Marek
teor
Alfredo Garcia