* updates book/../state-db-upgrades.md
* updates zebra-consensus with the changes from PR #9432 and minor test changes from #9295
* updates zebra-chain with the changes from PR #9432
* updates imports in zebra-rpc
* Merges and applies changes to zebra-state from PRs #9295 and #9432
* adds a TODO.
* minor update to zebra-rpc to use the new ReadResponse::AddressBalance type
* renames add_block_info db upgrade
* moves migration from `add_balance_received` to `block_info_and_address_received`
* replaces `write` with `write_batch`
* update disk format snapshots
* combines partial_transparent_balance_change() and partial_transparent_received_change()
* fixes typo
* Replaces redundant code with a call to `construct_column_families()`
* refactors `AddressBalanceLocation` and `AddressBalanceLocationChange` into a newtypes around a common inner type.
* fixes lints and doc compilation issue
* updates snapshots
* bumps the db format version to the next major version (27.0.0) instead of the next minor version (26.1.0)
* fixes clippy lint, minor cleanups
* fixes lint
* applies suggestions from code review
* fixes a mistake in `is_reusable_major_upgrade` where v26.0.0 is not a restorable db version
* Fixes an issue around marking an upgraded db format in the disk version file before completing migrations when reusing a previous major db format and restarting Zebrad before the migration is complete.
* Apply suggestions from code review
---------
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Update changelog
* Update end of support
* Add release date
* Add breaking change description
* Add new PRs to changelog
* Add breaking change description to changelog
* Update list of PRs in changelog
* chore: Release
* Apply suggestions from code review
Co-authored-by: Arya <aryasolhi@gmail.com>
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* refactor(docker): Enhance Zebra configuration options and entrypoint logic
- Introduced multiple methods for configuring Zebra, including custom config file paths and automatic generation from environment variables.
- Updated entrypoint script to handle configuration more robustly, checking for existing files and generating defaults as needed.
- Improved documentation in `docker.md` to clarify configuration options and their precedence.
- Adjusted environment variable handling in `docker/.env` for better clarity and functionality.
- Refactored `prepare_conf_file` to create a complete configuration based on environment variables, streamlining the setup process.
* fix(entrypoint): use correct `ZEBRA_RPC_PORT` variable
This also removes the logic to set the default port for RPC as this was introducing a buggy logic
* fix(docker): remove non-essential variables and set correct defaults
* chore(docker): use the default config for docker-compose as the default approach
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
* chore: better comment for user understanding
* fix(entrypoint): set permissions correctly for parent directories
* chore: align documentation and implementation
* chore: clearer and aligned comments with docker-compose
* fix(entrypoint): chown the zebra config file when `ZEBRA_CONF_PATH` is not set
---------
Co-authored-by: Marek <mail@marek.onl>
* refactor!: remove experimental features from release builds
BREAKING CHANGE: Experimental features are no longer built into release images.
Users must now build custom images with specific features if needed.
- Remove experimental build job from release workflow
- Remove experimental features from CI matrix
- Update documentation to reflect new feature strategy
- Simplify Dockerfile by removing experimental feature args
- Keep RUN_ALL_EXPERIMENTAL_TESTS for testing purposes only
Migration: Users who need specific features should:
1. Build custom images using --build-arg FEATURES="feature1 feature2"
2. Use the documented feature flags in their builds
3. Refer to the new documentation for building with custom features
Closes #9210Closes#9204
Partially #7415
* revert(ci): Add support for experimental features in test runs
Enable passing experimental features to Docker test runs by:
- Updating workflow configuration to pass EXPERIMENTAL_FEATURES
- Modifying entrypoint script to include experimental features during test execution
- Documenting new experimental test category in workflow README
* refactor(docs): streamline Docker documentation and feature testing strategy
- Reorganized and clarified advanced usage instructions for building Docker images with custom features, including metrics support.
- Enhanced the presentation of alternatives and configuration details for Docker usage.
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
* chore: revert deleted sentence
* Update book/src/user/docker.md
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
* Switch to a non-privileged user in tests
* Change test env setup
* Remove unneeded ARGs
* Simplify UID & GID handling in `runtime` target
* Simplify docs
* refactor(docker): Improve user and permission handling in Dockerfiles
- Add gosu for flexible non-root user execution
- Enhance user and group creation with configurable UID/GID
- Modify entrypoint script to support dynamic user switching
- Improve cache and log directory permission management
- Update comments to clarify user and permission strategies
* refactor(docker): Improve Zebra config file handling in entrypoint script
- Enhance error handling for missing config file (now exits with error)
- Simplify config preparation logic by removing redundant file copying
- Update comments to reflect new config file handling approach
- Ensure consistent use of ZEBRA_CONF_PATH throughout the script
* refactor(docker): Enhance container user security and configuration
- Increase UID/GID to 10001 to minimize host system user conflicts
- Remove `--system` flag from user and group creation to prevent potential issues
- Add detailed comments explaining UID/GID selection rationale
- Improve security by using high UID/GID values to reduce namespace collision risks
- Remove redundant `chmod` for entrypoint script
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Gustavo Valverde <g.valverde02@gmail.com>
Co-authored-by: Gustavo Valverde <gustavo@iterativo.do>
* Rm `.env` files
* Update `mining-docker.md`
* Revert "Rm `.env` files"
This reverts commit caaa4559c3.
* Add `enable_cookie_auth` to default Zebra conf
* Rename `default_zebra_config.toml`
* fmt `prometheus.yaml`
* Update `docker/test.env`
* Update `docker/.env`
* Refactor `docker compose` for lwd
* Enable disabling cookie authentication
* Update `docker compose` for tests
* Update general `docker compose`
* Update docs for running Zebra in Docker
* Add example `docker compose` file for Grafana
* Fix a bug in an example command
* Refactor test execution logic in entrypoint
* Rename `v2.1.0.toml` conf to `custom-conf.toml`
* Fix CI tests for loading of custom conf files
* Use the new conf file name in CI checks
* Use an extended regexp for custom conf CI check
* Refactor formatting & docs
* Refactor the `runtime` stage in Dockerfile
* Remove unused code from `entrypoint.sh`
* Simplify `entrypoint.sh` setup
* Revise docs & formatting
* Adjust default values for env vars
* Bump Rust v from 1.79 to 1.81 in Dockerfile
* Refactor `entrypoint.sh`
* Refactor `Dockerfile`
* Add TODOs for monitoring stage to Dockerfile
* Refactor `Dockerfile`
* Add TODOs for monitoring stage to Dockerfile
* Fix a typo
* Allow running `zebrad` in test mode
* Allow custom config for `zebrad` in test mode
* Remove `curl` from the `runtime` Docker image
* Remove redundant echos
* Remove a malfunctioning CD test
The test was using a custom config file set in `test_variables`.
However, the file was not included in the Docker image, and the
entrypoint script created a new, default one under the original file's
path. Zebra then loaded this new file, and the test passed because the
pattern in `grep_patterns` matched Zebra's output containing the
original path, even though the config file was different.
* Remove a redundant CI test
* Remove all packages from the `runtime` stage
* Docs cosmetics
* Clarify docs
* Bump Rust version
* Remove a security note
* Explicitly specify network cache dir
* Explicitly specify cookie dir
* Set UID, GID and home dir for the `zebra` user
* Set a working dir for the `zebra` user
* Don't remove `FEATURES`
* Try re-introducing the `testnet-conf` check
* `ZEBRA_CACHED_STATE_DIR` -> `ZEBRA_CACHE_DIR`
This dir doesn't hold only the state cache anymore, but also the cache
for network peers, and the cookie file.
* Refactor the dir structure
* Check that `ZEBRA_CONF_PATH` exists in the image
* Improve the check for `ZEBRA_CONF_PATH`
* Use different flag in the `ZEBRA_CONF_PATH` check
* Simplify the `ZEBRA_CONF_PATH` check
* Fix spelling
* Comment out the `testnet-conf` CI check
* Add commented out `test-zebra-conf-path` CI check
* Reintroduce `testnet-conf` CI check
* Update the `custom-conf` CI check
* Add `v2.1.0.toml` conf file
* Refine the `v2.1.0.toml` conf file
* Remove `ZEBRA_LISTEN_ADDR` from the entrypoint
* Remove `ZEBRA_CHECKPOINT_SYNC` from the entrypoint
* Stop supporting configuration of the RPC port
* Add default conf file
* Prepare Zebra's config in the entrypoint script
* Remove unneeded packages from the `deps` target
* Docs cosmetics
* Use only `$FEATURES` in entrypoint
* Simplify handling of Rust features
* Add a TODO
* Add CI debug statements
* Don't require test vars in conf test
* Reintroduce `protoc`
* Remove `-e NETWORK`
* Remove `ZEBRA_FORCE_USE_COLOR=1`
* Remove `ZEBRA_CACHE_DIR=/var/cache/zebrad-cache`
* Reintroduce the "custom-conf" test
* Set up test env the same way as prod
* Don't repeatedly check for conf file in entrypoint
* Simplify file ownership in Dockerfile
* Fix checkpoint tests in entrypoint
* Fix Zebra config CI tests
* `LIGHTWALLETD_DATA_DIR` -> `LWD_CACHE_DIR`
* Add config for `LWD_CACHE_DIR` to Dockerfile
* `/var/cache/zebrad-cache` -> `~/.cache/zebra`
* `var/cache/lwd-cache` -> `/home/zebra/.cache/lwd`
* Remove `LOG_COLOR=false` from GCP setup
* Don't specify `LWD_CACHE_DIR` in CI tests
* Don't switch to `zebra` user for tests in Docker
* Join "experimental" and "all" tests in CI
* Remove outdated docs
* Refactor tests with fake activation heights
* Fix tests for scanner
* Remove unnecessary clone
* Fix test in zebra-script
* Document that RPC server addr must be set
* Change the log level of missing port msg to info
Having the RPC port not specified is part of Zebra's normal operation,
and is the default config.
* Fix links in docs for tracing
* Update zebrad/src/commands/start.rs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Adds new column family for [spent_out_loc] -> [spending_tx_loc] with a read method and an update to `prepare_spending_transparent_tx_ids_batch()` for maintaining it when committing blocks to the finalized state.
Adds TODOs for remaining production changes needed for issue #8837.
* add spending tx ids for spent outpoints to non-finalized chains
* adds a `spending_transaction_hash()` read fn for the new column family
* Adds a `TransactionIdForSpentOutPoint` ReadRequest and a `TransactionId` ReadResponse
* Updates snapshots, removes outdated TODOs, moves a TODO.
* Clarifies `spent_utxos` field docs, fixes an assertion
* import TypedColumnFamily from `finalized_state` instead of from the crate.
* adds db format upgrade for spent outpoints -> tx hash
* adds revealing tx ids for nullifiers in finalized and non-finalized states
* updates nullifiers column families to include revaling transaction locations in db format upgrade
* Renames new read state request to `SpendingTransactionId` and updates its type to a `Spend` enum
* refactor db format upgrade and prepare_nullifiers_batch() to use ZebraDb instead of DiskDb, checks cancel_receiver before every db operation
* Adds acceptance test for checking that the finalized state has spending transaction ids
* Adds variant docs to zebra_state::request::Spend enum
* Updates Zebra book with the latest changes to the rocks db column families
* Updates acceptance test to check non-finalized state
* adds a few log messages to the acceptance test, reduces frequency of logs for progress updates
* fixes docs lint and skips test when there is no cached state
* Avoids returning genesis coinbase tx hash when indexes are missing
* Adds `indexer` compilation feature in zebra-state and build metadata in db format version file
* stops tracking new indexes in finalized state when feature is unselected
* stops tracking new indexes in non-finalized state when indexer feature is unselected
* condenses imports
* - adds build metadata when writing db version file, if any.
- adds the build metadata to the db version file before adding indexes.
- deletes indexes when running without the `indexer` feature
* Replaces dropping cf with deleting range of all items to avoid a panic when trying to open the db with that column family.
* Fixes lint, avoids reading coinbase transactions from disk
* updates db column families table
* Document need for having an indexed cached state and use a multi-threaded tokio runtime in has_spending_transaction_ids test
* fixes call to renamed `future_blocks` test fn
* improves test logs and fixes a disk format deserialization bug
* Replaces a new expr with a previously existing constant, fixes typo
* repeat words
* Update zebra-chain/src/block/serialize.rs
Co-authored-by: Pili Guerra <1311133+mpguerra@users.noreply.github.com>
---------
Co-authored-by: Pili Guerra <1311133+mpguerra@users.noreply.github.com>
* book: add section about private testnet testing
* Apply suggestions from code review
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Arya <aryasolhi@gmail.com>
---------
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
Co-authored-by: Arya <aryasolhi@gmail.com>
* update documentation related auth cookie method for rpcs in the zebra book
* Update book/src/user/lightwalletd.md
* update rpc auth statement
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Updates `EOS_PANIC_AFTER` end of support constant from 16 weeks to 14 weeks, so the Mainnet panic height is just before the second halving and end of the current dev fund.
* version bumps
* Updates CHANGELOG.md
* Updates changelog
* updates end of support time to 10 weeks
* Adds a note to the changelog about recovering after finalizing blocks from a fork
* Add new changes on main to CHANGELOG.md
* Updates ESTIMATED_RELEASE_HEIGHT
* Apply suggestions from code review
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* Update zebrad/src/components/sync/end_of_support.rs
---------
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* validate zebrad cache dir
* update scanner documentation
* clippy
* Apply suggestions from code review
Co-authored-by: Arya <aryasolhi@gmail.com>
* add info level by default to scanner application
* remove `RUST_LOG=info` from some of the test examples on how to run
* remove `RUST_LOG=info` from the book sample
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Update `CHANGELOG.md`
* chore: Release
* Update versions in the release dry-run workflow
* Update EOS
* Estimate release height to ~ upcoming Tuesday
* Apply suggestions from code review
Co-authored-by: Arya <aryasolhi@gmail.com>
* Add a "Summary" title to the changelog
* Remove the summary title
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Move "Known Issues" from README to the book
* Remove an old note from the book
* Move optional configs & features to the book
* Fix TOC in `README.md`
* Refactor the section on installing Zebra
* Fix badges in `README.md`
* Mention optional features and troubleshooting
* Adds documention to the Zebra book about custom Testnets, allows for configuring custom Testnet genesis hashes, refactors check for compatible custom Testnets, adds a TODO, and uses the default testnet when there's an empty or default `testnet_parameters` field in Zebra's network config.
* Adds example configs to custom Testnet docs, moves note about inbound connections on Regtest to a footnote.
* Adds a changelog entry
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
* Addresses suggestions from code review.
* Apply suggestions from code review
Co-authored-by: Arya <aryasolhi@gmail.com>
* Update book/src/user/custom-testnets.md
---------
Co-authored-by: Marek <mail@marek.onl>
* Always set mempool.debug_enable_at_height to 0 on Regtest
* Adds a Regtest with Zebra page to the book
* Deduplicates calls to is_regtest() in start fn
* upgrade min rust version in zebrad
* Display full network name in logs except for Mainnet and default Testnet
* Add example logs/code to regtest section of the book
* apply some extra information
Co-authored-by: Arya <aryasolhi@gmail.com>
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
* Reduce the end of support time from 20 weeks to 16 weeks.
* Update release-process section of the book
* Adds changelog entry
* Update CHANGELOG.md
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* changelog and readme for 1.7.0 release
* chore: Release
* update zebra-scan and zebra-grpc to the release crates dry run script
* update ESTIMATED_RELEASE_HEIGHT
* update the release period to 6 weeks
* update release block
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
* Apply suggestions from code review
Co-authored-by: Marek <mail@marek.onl>
---------
Co-authored-by: Marek <mail@marek.onl>
Arya
Marek
Gustavo Valverde
VolodymyrBg
crStiv
cryptoraph
Alfredo Garcia
futreall
Dimitris Apostolou
root
Cypher Pepe
pinglanlu
Conrado Gouvea
ying tong