* Fix the combining of Sapling subtrees
This commit fixes a bug due to which the function `sapling_subtrees`
used to always include all subtrees from the non-finalized state without
respecting the given limit.
* Fix the combining of Orchard subtrees
This commit fixes a bug due to which the function `orchard_subtrees`
used to always include all subtrees from the non-finalized state without
respecting the given limit.
* Add additional checks when retrieving subtrees
* Allow raw subtree insertions into `Chain` in tests
* Derive `Default` for `Chain`
* Derive `Default` for note commitment tree nodes
* Use `pub(super)` for `DiskWriteBatch`
Set the visibility of `DiskWriteBatch` to `pub(super)`.
* Add tests for retrieving subtrees
* Use `default()` for subtree roots in tests
This change is unrelated to the PR.
* Refactor docs for inserting subtrees
This change is unrelated to the PR.
* Add a progress bar config that is disabled unless the feature is on
* Simplify the default config
* Enable the progress bar feature by default, but require the config
* Rename progress bars config to avoid merge conflicts
* Use a log file when the progress bar is activated
* Document how to configure progress bars
* Handle log files in config_tests and check config path
* Fix doc link
* Fix path check
* Fix config log matching
* Fix clippy warning
* Add tracing to config tests
* It's zebrad not zebra
* cargo fmt --all
* Update release for config file changes
* Fix config test failures
* Allow printing to stdout in a method
* ref(docker): combine test and prod entrypoints into one
This makes the following changes
- Uses `-x` instead of echoing the variables values
- Sets default values where required
- Create a function to list directories
- Create a function to run cargo tests
- Use a better approach to handle different options in the `case` manegement for tests and production
- Replaces all instances of `runtime-entrypoint.sh` with `entrypoint.sh`
* chore: use 2 spaces insted of 4 for indentation
* fix: add missing `test-threads` to `fully_synced_rpc_`
* fix: allow 3 arguments instead of 2 to handle `fully_synced_rpc_` correctly
* chore: remove extra file
* chore: add comments to main parts of the file
* fix(ci): just create the `$ZEBRA_CONF_PATH` if `$ENTRYPOINT_FEATURES` is not set
* fix(entrypoint): just print ${ZEBRA_CONF_PATH} if exists
* fix: missing condition
* ref: handle tests better if `$ENTRYPOINT_FEATURES` is set
* fix(ci): We just want `ZEBRA_CONF_PATH` to be set in the `release` image
* fix(entrypoint): fix the overall `case` logic
* fix(ci): allos to run with custom config in CI image
* fix(ci): more edgecases
* fix: we don't need to find files, but subdirectories
* fix(ci): handle Signal Forwarding and exit codes for `cargo`
* fix(ci): parse `fully_synced_rpc_` test correctly
* chore: add missing cache dir variable for LWD
* fix(entrypoint): handle an arbitrary number of arguments
* fix(entrypoint): handle features list
* fix(entrypoint): typo
* chore: typo
* Handle negative and zero getnetworksolsps arguments correctly
* Simplify the solsps loop structure
* Simplify loop by avoiding manual iteration and peeking
* Avoid division by zero
* Use min and max times rather than first and last times
* Refactor block iterators so they are more efficient
* Make finding chains easier
* Simplify block iteration code
* Remove implemented TODO comments
* Simplify internal iterator state
* Implement iteration by any chain item
* Iterate block headers rather than full blocks
* Ignore code that is (sometimes) dead
* Fix a dead code warning
* Add a no blocks in state error constant
* Check result values in the RPC test
* Fix invalid calculation handling
* fix: use `exit-nopipe` with consistent `shell` usage
Temporarily disabled the `set -e` option around the docker logs command to handle the broken pipe error gracefully.
Handle more complex scenarios in our `Result of ${{ inputs.test_id }} test` job
* fix: Use single quotes for the outer command
* fix: use same approach for CD
* test: check launch failure logs
* fix: revert CD changes
* fix: do not try to increase the disk size and wait mounting
* fix: increase GB a bit more
* fix: do not fail on pipe failure
* fix: use plain `tee /dev/stderr`
If this does not work try `(tee … || true)`
* fix: `tee` not stoping on cd config tests
* fix: match logic with GCP tests
* fix(cd): handle pipe and other errors correctly
* try `tee --output-error=exit-nopipe`
* fix: TRAP without pipefail
* test: pipefail with exit and trap
* fix: use a subshell
* fix(ci): wait for mounting and show system logs if fail
* fix(ci): GCP is not always mounting disks in the same order
* fix: use `grep` instead of `awk`
* fix: typo
* fix: use simpler `grep` command
* fix: do not sleep if not require
* chore: reduce diff
* add `sapling_subtree_for_tip` and `orchard_subtree_for_tip` methods to `ZebraDb`
* add methods for non finalized state, move functions
* call `zs_last_key_value` the right way
* fix and simplify `*_subtree_for_tip` methods
Co-authored-by: Arya <aryasolhi@gmail.com>
* apply filter
* rename all tree and subtree methods that use tip
* rename tip tree and subtree methods in non finalized chain
* apply simplify suggestions
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: teor <teor@riseup.net>
* add buld failures ticket generation to most important workflows
* add missing newline at the end of the file
* doc fixes
* add missing testnet build check
Co-authored-by: teor <teor@riseup.net>
---------
Co-authored-by: teor <teor@riseup.net>
* fix(ci): disk validation for docker volume mount
* Use a symlink for lightwalletd cached state rather than mounting the same volume twice
* Avoid "sdb seems to be busy" errors from docker by adding extra sleeps
* Add a missing backslash
* Remove symlink from workflow
* Symlink lightwalletd path in entrypoint.sh
* Retry on failure and check Docker logs
* End ssh shell lines with explicit terminators
* Delete Docker containers if Docker mount fails
* Revert symlink changes in entrypoint.sh
* Debug using lsof
* Use correct lsof commands
* Use correct syntax for lsof +D
* fix(ci): make multiple validations before mounting
Loop and checks for three conditions:
The device `/dev/sdb` exists.
No process is using the device `/dev/sdb`.
No process is using the Docker volume directory.
* fix: do not pre-mount docker volume
The Docker version available with the newer `cos-stable` OS (https://cloud.google.com/release-notes#cos-109-17800-0-45) allows to mount the image when running it.
Mounting it before makes the disk unavailable.
* fix: remove extra `;`
* fix: just confirm with `lsof` and show it's output
* chore: reduce diff
---------
Co-authored-by: teor <teor@riseup.net>
* Add a state full sync to the release-checklist.md
* Add state test and freeze to release.md
* Make testing requirement clearer
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Provide a user hint when the database format is invalid
* Split a path-based database version method
* Check the database format before Zebra shuts down
* Fix a typo in zebra-state/src/service/finalized_state/zebra_db.rs
Co-authored-by: Marek <mail@marek.onl>
* Fix a typo in upgrade.rs
---------
Co-authored-by: Marek <mail@marek.onl>
* Refactor format checks, stop running quick check before upgrade
* Speed up startup by always running the format checks in parallel
* Make detailed format checks cancellable on shutdown
* Prepare for periodic format checks
* Time upgrades and validity checks
* Run a database format check every 5 minutes
* Wait for a cancel signal rather than unconditionally sleeping
* Move check_max_on_disk_height() into the format checks
* Move spawn_format_change() into its own method
* Avoid manual handling of previous sapling trees by using iterator windows instead
* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead
* Simplify adding notes by using the exact number of remaining notes
* Simplify by skipping the first block, because it can't complete a subtree
* Re-use existing tree update code
* Apply the sapling changes to orchard subtree updates
* add a reverse database column family iterator function
* Make skipping the lowest tree independent of iteration order
* Move new subtree checks into the iterator, rename to end_height
* Split subtree calculation into a new method
* Split the calculate and write methods
* Quickly check the first subtree before running the full upgrade
* Do the quick checks every time Zebra runs, and refactor slow check error handling
* Do quick checks for orchard as well
* Make orchard tree upgrade match sapling upgrade code
* Upgrade subtrees in reverse height order
* Bump the database patch version so the upgrade runs again
* Reset previous subtree upgrade data before doing this one
* Add extra checks to subtree calculation to diagnose errors
* Use correct heights for subtrees completed at the end of a block
* Add even more checks to diagnose issues
* Instrument upgrade methods to improve diagnostics
* Prevent modification of re-used trees
* Debug with subtree positions as well
* Fix an off-by-one error with completed subtrees
* Fix typos and confusing comments
Co-authored-by: Marek <mail@marek.onl>
* Fix mistaken previous tree handling and end tree comments
* Remove unnecessary subtraction in remaining leaves calc
* Log heights when assertions fail
* Fix new subtree detection filter
* Move new subtree check into a method, cleanup unused code
* Remove redundant assertions
* Wait for subtree upgrade before testing RPCs
* Fix subtree search in quick check
* Temporarily upgrade subtrees in forward height order
* Clarify some comments
* Fix missing test imports
* Fix subtree logging
* Add a comment about a potential hang with future upgrades
* Fix zebrad var ownership
* Log more info when add_subtrees.rs fails
* cargo fmt --all
* Fix unrelated clippy::unnecessary_unwrap
* cargo clippy --fix --all-features --all-targets; cargo fmt --all
* Stop the quick check depending on tree de-duplication
* Refactor waiting for the upgrade into functions
* Wait for state upgrades whenever the cached state is updated
* Wait for the testnet upgrade in the right place
* Fix unused variable
* Fix a subtree detection bug and comments
* Remove an early reference to reverse direction
* Stop skipping subtrees completed at the end of blocks
* Actually fix new subtree code
* Upgrade subtrees in reverse height order
Reverts "Temporarily upgrade subtrees in forward height order"
This reverts commit a9558be214.
* Bump the database patch version to re-run the upgrade (for testing)
* Revert "Remove an early reference to reverse direction"
This reverts commit c206404377.
---------
Co-authored-by: Marek <mail@marek.onl>
dependabot[bot]
Marek
teor
Arya
Gustavo Valverde
Alfredo Garcia