* Fix GitHub workflows
There are two workflows that have bugs in them:
- cd-deploy-nodes-gcp is missing a job name, which is then depended-on
later
- sub-ci-integration-tests-gcp changed the name of a job without
changing its dependents
* Stop certain CI jobs from running on forks
There are various GitHub CI jobs that won’t work on forks. E.g., some
need credentials for ZF’s DockerHub or GCP accounts.
Unfortunately, you can’t prevent entire workflows from running this way,
but this disables a minimal number of jobs to keep forks from failing
whenever `main` is pushed to.
* introduce new fields to `GetInfo`
* add address book to methods for getinfo connections field
* add the version field to getinfo
* track last node error or warning and display it in getinfo rpc method
* add the rest of the fields, minor cleanup
* fix the tests
* clippy
* feat(network): Add misbehavior tracking for peers in the address book
* - Add a `misbehavior_score` field to `MetaAddr`
- Add a `bans_by_ip` field to `AddressBook`
- Update the `AddressBook::update()` method to:
- increment misbehavior scores in its entries,
- add addr ips to bans_by_ip if the score is excessive,
- remove any addrs at the banned ip
- Avoid responding to `GetAddr` requests with addresses of misbehaving peers (return None from `sanitized()`),
- Avoid new inbound or outbound connections to banned ips
* Drops banned peer connections in peer set's `poll_ready()` method
* Adds rudimentary misbehavior score tracking
* fixes some proptests, moves and removes some TODOs
* fixes lint
* Removes outdated TODO
* Adds stub for acceptance test
* updates call to updated fn
* Stores likely inbound peer connection addresses in address book and return their IPs with the default port instead of the transient port when responding to GetAddr requests
* Avoids gossiping peer addrs from inbound connections
* updates test to check that sanitize won't return inbound peer addrs or addrs with non-zero misbehaviour scores.
updated misbehaviour score for potentially unavoidable errors.
* Updates `generate` RPC to support any network where PoW is disabled.
Updates acceptance test to check that zebrad instances disconnect once one of them advertises a block with an invalid PoW
* minor tangential cleanup
* Finishes acceptance test, sends misbehavior updates from syncer, and always updates address book entries if the update is to their misbehaviour score
* skip test on windows (to minimize risk of port conflict)
* Applies suggestions from code review
This change enhances the GCP deployment workflows with several improvements:
- Add static IP address support for long-running nodes
- Implement dynamic network selection matrix
- Add manual deployment trigger support
- Update GCP configuration syntax for better compatibility
- Remove redundant deploy-instance job
- Improve environment labeling and log file handling
The static IP addresses will help maintain stable network connectivity
for long-running nodes, particularly important for testnet peer discovery.
Fixes#8763
Release Drafter v6.1.0 has a regression that creates multiple duplicate draft
releases instead of updating existing ones. This change:
- Disables pre-release flag for draft releases
- Excludes pre-releases from being included in drafts
- Explicitly defines PR event types in workflow
See: release-drafter/release-drafter#1425
* Refactor formatting & docs
* Refactor the `runtime` stage in Dockerfile
* Remove unused code from `entrypoint.sh`
* Simplify `entrypoint.sh` setup
* Revise docs & formatting
* Adjust default values for env vars
* Bump Rust v from 1.79 to 1.81 in Dockerfile
* Refactor `entrypoint.sh`
* Refactor `Dockerfile`
* Add TODOs for monitoring stage to Dockerfile
* Refactor `Dockerfile`
* Add TODOs for monitoring stage to Dockerfile
* Fix a typo
* Allow running `zebrad` in test mode
* Allow custom config for `zebrad` in test mode
* Remove `curl` from the `runtime` Docker image
* Remove redundant echos
* Remove a malfunctioning CD test
The test was using a custom config file set in `test_variables`.
However, the file was not included in the Docker image, and the
entrypoint script created a new, default one under the original file's
path. Zebra then loaded this new file, and the test passed because the
pattern in `grep_patterns` matched Zebra's output containing the
original path, even though the config file was different.
* Remove a redundant CI test
* Remove all packages from the `runtime` stage
* Docs cosmetics
* Clarify docs
* Bump Rust version
* Remove a security note
* Explicitly specify network cache dir
* Explicitly specify cookie dir
* Set UID, GID and home dir for the `zebra` user
* Set a working dir for the `zebra` user
* Don't remove `FEATURES`
* Try re-introducing the `testnet-conf` check
* `ZEBRA_CACHED_STATE_DIR` -> `ZEBRA_CACHE_DIR`
This dir doesn't hold only the state cache anymore, but also the cache
for network peers, and the cookie file.
* Refactor the dir structure
* Check that `ZEBRA_CONF_PATH` exists in the image
* Improve the check for `ZEBRA_CONF_PATH`
* Use different flag in the `ZEBRA_CONF_PATH` check
* Simplify the `ZEBRA_CONF_PATH` check
* Fix spelling
* Comment out the `testnet-conf` CI check
* Add commented out `test-zebra-conf-path` CI check
* Reintroduce `testnet-conf` CI check
* Update the `custom-conf` CI check
* Add `v2.1.0.toml` conf file
* Refine the `v2.1.0.toml` conf file
* Remove `ZEBRA_LISTEN_ADDR` from the entrypoint
* Remove `ZEBRA_CHECKPOINT_SYNC` from the entrypoint
* Stop supporting configuration of the RPC port
* Add default conf file
* Prepare Zebra's config in the entrypoint script
* Remove unneeded packages from the `deps` target
* Docs cosmetics
* Use only `$FEATURES` in entrypoint
* Simplify handling of Rust features
* Add a TODO
* Add CI debug statements
* Don't require test vars in conf test
* Reintroduce `protoc`
* Remove `-e NETWORK`
* Remove `ZEBRA_FORCE_USE_COLOR=1`
* Remove `ZEBRA_CACHE_DIR=/var/cache/zebrad-cache`
* Reintroduce the "custom-conf" test
* Set up test env the same way as prod
* Don't repeatedly check for conf file in entrypoint
* Simplify file ownership in Dockerfile
* Fix checkpoint tests in entrypoint
* Fix Zebra config CI tests
* `LIGHTWALLETD_DATA_DIR` -> `LWD_CACHE_DIR`
* Add config for `LWD_CACHE_DIR` to Dockerfile
* `/var/cache/zebrad-cache` -> `~/.cache/zebra`
* `var/cache/lwd-cache` -> `/home/zebra/.cache/lwd`
* Remove `LOG_COLOR=false` from GCP setup
* Don't specify `LWD_CACHE_DIR` in CI tests
* Don't switch to `zebra` user for tests in Docker
* Join "experimental" and "all" tests in CI
* Remove outdated docs
* Refactor tests with fake activation heights
* Fix tests for scanner
* zebra-rpc: Correctly map JSON-RPC to/from 2.0
The existing code was only handling the `lightwalletd` client format
which includes a `"jsonrpc": "1.0"` entry, and it was returning
responses as JSON-RPC 2.0 regardless of what the client expected.
* Update zebra-rpc/src/server/http_request_compatibility.rs
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
* Adds some of the required fields on `getblockchaininfo` output.
* Adds state request/response variants for querying disk usage
* Adds `size_on_disk`, `chain_supply`, and `monitored` fields.
* Updates snapshots
* fixes prop tests
* fixes doc lints
* Adds missing `size()` method
* Fixes lwd integration test issue by updating get_blockchain_info to fallback on default values instead of returning an error if the state is empty.
Related: Runs state queries in parallel from getblockchaininfo RPC and removes the BlockHeader query by getting the tip block time from the latest chain tip channel.
* Updates failing proptests
* fixes lint
* Adds new invalidate_block method to non finalized state. Adds test case in vectors.rs. Updates non finalized state to track invalidated_blocks
* Removes InvalidatedBlockData struct. Wraps invalidated_blocks HashMap values with Arc. Optimizies invalidate_block fn to return early if root has hash of desired block.
* Update metrics whenever chain_set is modified
* Adds child_blocks method and refactors invalidate_block in chain.rs. Refactors invalidate_block in NonFinalizedState.
---------
Co-authored-by: Elijah Hampton <elijahhampton@pop-os.lan>
Release Drafter v6.1.0 introduced a regression that causes it to create
multiple duplicate draft releases instead of updating existing ones. This
was causing ~160 duplicate draft releases to be created for Zebra 2.2.0.
Pin to v6.0.0 which correctly finds and updates existing draft releases.
This is a known issue tracked in release-drafter/release-drafter#1425.
* add a channel for submit_block notifications to gossip task
* fix tests and gossip logic
* remove the network discriminant and add a test
* clippy suggestions
* fix unused variable
* attempt to fix the conditional compilation issues
* fix default
* Suggestions for "fix(mining): Advertise mined blocks" (#9183)
* refactor error conversions in GetBlockTemplateRpcImpl and rewords documentation
* Replaces polling mined block receiver with a select
* Skip checking that Zebra is likely synced to the network tip before returning block templates on Testnet.
* fixes a clippy lint and a concurrency bug
---------
Co-authored-by: Arya <aryasolhi@gmail.com>
## Motivation
Our CI/CD infrastructure is complex and was lacking proper documentation, making it challenging for newcomers and contributors to understand the system. This PR adds comprehensive documentation to:
- Make our CI/CD architecture more accessible to new contributors
- Document workflow relationships and dependencies
- Explain infrastructure components and their roles
- Provide best practices and known limitations
Closes#9112
### Specifications & References
- [GitHub Actions Path Filtering Limitation](https://github.com/orgs/community/discussions/44490)
- [GitHub Actions Variables in Forks Limitation](https://github.com/orgs/community/discussions/44322)
- [Rust Caching Limitations](https://github.com/ZcashFoundation/zebra/issues/6169#issuecomment-1712776391)
## Solution
1. Added detailed CI/CD documentation in `.github/workflows/README.md`:
- Table of contents for easy navigation
- System overview and core infrastructure details
- Mermaid diagram showing workflow relationships and parallel execution
- Detailed sections on:
- Core infrastructure components
- Workflow organization
- Test execution strategy
- Infrastructure details
- Best practices
- Known issues
2. Added CI/CD section to main `README.md`:
- Brief overview of CI/CD capabilities
- Strategic placement between Getting Started and Documentation sections
- Direct link to detailed documentation
3. Documentation improvements:
- Clear explanation of patch workflows and their rationale
- Detailed infrastructure dependencies and requirements
- Comprehensive coverage of test execution patterns
- External contributor considerations and limitations
* Remove unnecessary clone
* Fix test in zebra-script
* Document that RPC server addr must be set
* Change the log level of missing port msg to info
Having the RPC port not specified is part of Zebra's normal operation,
and is the default config.
* Fix links in docs for tracing
* Update zebrad/src/commands/start.rs
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
---------
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Wait for spent UTXOs to arrive in the state if they're not found elsewhere instead of returning an error when a transaction in a candidate block has already been verified in the mempool
* updates test, avoids returning InternalDowncastErrors when AwaitUtxo requests time out
* return early if there are no missing deps
* Applies suggestions from code review
* Updates field documentation
* Adds new column family for [spent_out_loc] -> [spending_tx_loc] with a read method and an update to `prepare_spending_transparent_tx_ids_batch()` for maintaining it when committing blocks to the finalized state.
Adds TODOs for remaining production changes needed for issue #8837.
* add spending tx ids for spent outpoints to non-finalized chains
* adds a `spending_transaction_hash()` read fn for the new column family
* Adds a `TransactionIdForSpentOutPoint` ReadRequest and a `TransactionId` ReadResponse
* Updates snapshots, removes outdated TODOs, moves a TODO.
* Clarifies `spent_utxos` field docs, fixes an assertion
* import TypedColumnFamily from `finalized_state` instead of from the crate.
* adds db format upgrade for spent outpoints -> tx hash
* adds revealing tx ids for nullifiers in finalized and non-finalized states
* updates nullifiers column families to include revaling transaction locations in db format upgrade
* Renames new read state request to `SpendingTransactionId` and updates its type to a `Spend` enum
* refactor db format upgrade and prepare_nullifiers_batch() to use ZebraDb instead of DiskDb, checks cancel_receiver before every db operation
* Adds acceptance test for checking that the finalized state has spending transaction ids
* Adds variant docs to zebra_state::request::Spend enum
* Updates Zebra book with the latest changes to the rocks db column families
* Updates acceptance test to check non-finalized state
* adds a few log messages to the acceptance test, reduces frequency of logs for progress updates
* fixes docs lint and skips test when there is no cached state
* Avoids returning genesis coinbase tx hash when indexes are missing
* Adds `indexer` compilation feature in zebra-state and build metadata in db format version file
* stops tracking new indexes in finalized state when feature is unselected
* stops tracking new indexes in non-finalized state when indexer feature is unselected
* condenses imports
* - adds build metadata when writing db version file, if any.
- adds the build metadata to the db version file before adding indexes.
- deletes indexes when running without the `indexer` feature
* Replaces dropping cf with deleting range of all items to avoid a panic when trying to open the db with that column family.
* Fixes lint, avoids reading coinbase transactions from disk
* updates db column families table
* Document need for having an indexed cached state and use a multi-threaded tokio runtime in has_spending_transaction_ids test
* fixes call to renamed `future_blocks` test fn
* improves test logs and fixes a disk format deserialization bug
* Replaces a new expr with a previously existing constant, fixes typo
* Refactors getblockchaininfo to return gensis block if tip pool request fails. Adds test in prop.rs
* clippy, check and fmt
* Removes unused imports. Refactors code for correctness in genesis bytes reference and tokio pause() call
---------
Co-authored-by: Elijah Hampton <elijahhampton@pop-os.lan>
* replace jsonrpc-core with jsonrpsee-types in zebra-node-services
* remove non needed feature from dependency
* remove jsonrpc-core as a dev dependency in zebra-node-services
* add jsonrpsee-types as a dev dependency
- Renamed `no_cached_disk` input to `need_cached_disk` in `cd-deploy-nodes-gcp.yml` to clarify its purpose.
- Adjusted conditional logic for cached disk usage based on event types (release vs. non-release) in `cd-deploy-nodes-gcp.yml`.
- Removed the environment strategy matrix from `sub-deploy-integration-tests-gcp.yml` to simplify the workflow configuration.
- Enhanced comments for better clarity on caching and environment handling across workflows.
* repeat words
* Update zebra-chain/src/block/serialize.rs
Co-authored-by: Pili Guerra <1311133+mpguerra@users.noreply.github.com>
---------
Co-authored-by: Pili Guerra <1311133+mpguerra@users.noreply.github.com>
* feat(actions): use GitHub `environments` for infra deployments
* chore(workflows): enhance environment handling in CI/CD configurations
- Added dynamic environment assignment in `cd-deploy-nodes-gcp.yml` based on event type (release or dev).
- Updated `sub-build-docker-image.yml` to utilize the `inputs.environment` for environment configuration.
- Introduced a strategy matrix for environment selection in `sub-deploy-integration-tests-gcp.yml`, allowing for both dev and prod environments based on the branch.
- Ensured `sub-find-cached-disks.yml` uses the `inputs.environment` for consistency across workflows.
* fix(workflows): streamline environment input handling in CI configurations
- Removed the required environment input from `sub-ci-integration-tests-gcp.yml`.
- Updated comments in `sub-deploy-integration-tests-gcp.yml` to clarify the strategy for creating images in dev and prod environments based on the main branch.
* test: just set the environmet when diff to dev
* refactor(workflows): unify environment handling across CI configurations
- Removed the optional environment input from multiple workflow files, including `manual-zcashd-deploy.yml`, `sub-build-docker-image.yml`, and `sub-deploy-integration-tests-gcp.yml`.
- Updated environment assignment logic to consistently use 'prod' for release events across `cd-deploy-nodes-gcp.yml`, `sub-build-docker-image.yml`, `sub-find-cached-disks.yml`, and `sub-deploy-integration-tests-gcp.yml`.
- Enhanced clarity in comments regarding environment strategies in `sub-deploy-integration-tests-gcp.yml`.
* fix(workflows): update environment assignment logic for CI configurations
- Changed environment assignment in `cd-deploy-nodes-gcp.yml`, `sub-build-docker-image.yml`, and `sub-find-cached-disks.yml` to use 'dev' as a fallback when the event is not a release, to avoid a `false` fallback
Greg Pfeil
Arya
Alfredo Garcia
Conrado Gouvea
Marek
Gustavo Valverde
Jack Grigg
Metalcape
dependabot[bot]
Elijah Hampton
cryptoraph
Fallengirl
futreall
Pili Guerra
Dimitris Apostolou
root
Marius Kjærstad
idky137