Commit Graph

1319 Commits

Author SHA1 Message Date
Alfredo Garcia 5824f85bb4
Release v1.5.1 (#8165)
* add 1.5.1 changelog

* extend eos

* chore: Release

* add suggestions from review

Co-authored-by: Marek <mail@marek.onl>

* Apply suggestions from code review

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2024-01-19 21:14:32 +00:00
dependabot[bot] e7e3970bfd
build(deps): bump the prod group with 3 updates (#8158)
Bumps the prod group with 3 updates: [clap](https://github.com/clap-rs/clap), [vergen](https://github.com/rustyhorde/vergen) and [zcash_address](https://github.com/zcash/librustzcash).


Updates `clap` from 4.4.14 to 4.4.16
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.4.14...v4.4.16)

Updates `vergen` from 8.2.6 to 8.2.10
- [Release notes](https://github.com/rustyhorde/vergen/releases)
- [Commits](https://github.com/rustyhorde/vergen/compare/8.2.6...8.2.10)

Updates `zcash_address` from 0.3.0 to 0.3.1
- [Release notes](https://github.com/zcash/librustzcash/releases)
- [Commits](https://github.com/zcash/librustzcash/compare/zcash_address-0.3.0...zcash_address-0.3.1)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: vergen
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: zcash_address
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marek <mail@marek.onl>
2024-01-15 23:27:00 +00:00
Marek 01fa50cba3
change(doc): Refactor user & dev docs (#8145)
* Fix the rendering of expandable content

* Refactor the docs refs in the readme

* Change the search query for docs.rs

* Refactor the landing docs for the `zebrad` crate

* List Zebra crates in the landing docs for `zebrad`

* Remove `## Future Work` from the main readme

This section doesn't say anything useful.

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2024-01-12 22:36:04 +00:00
teor 54c702e73c
fix(miner): Avoid some duplicate block errors (#8150)
* Update to the latest equihash solver code

* Wait for a new template after mining a block

* Remove an unused import

* Another solver update
2024-01-12 06:19:05 +00:00
dependabot[bot] 5878b325d5
build(deps): bump the prod group with 5 updates (#8138)
Bumps the prod group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [clap](https://github.com/clap-rs/clap) | `4.4.11` | `4.4.14` |
| [semver](https://github.com/dtolnay/semver) | `1.0.20` | `1.0.21` |
| [thiserror](https://github.com/dtolnay/thiserror) | `1.0.51` | `1.0.56` |
| [serde_json](https://github.com/serde-rs/json) | `1.0.108` | `1.0.111` |
| [tempfile](https://github.com/Stebalien/tempfile) | `3.8.1` | `3.9.0` |


Updates `clap` from 4.4.11 to 4.4.14
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.4.11...v4.4.14)

Updates `semver` from 1.0.20 to 1.0.21
- [Release notes](https://github.com/dtolnay/semver/releases)
- [Commits](https://github.com/dtolnay/semver/compare/1.0.20...1.0.21)

Updates `thiserror` from 1.0.51 to 1.0.56
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.51...1.0.56)

Updates `serde_json` from 1.0.108 to 1.0.111
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.108...v1.0.111)

Updates `tempfile` from 3.8.1 to 3.9.0
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.8.1...v3.9.0)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: semver
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-11 16:38:44 +00:00
teor 2ac6921d60
feat(mine): Add an internal Zcash miner to Zebra (#8136)
* Patch equihash to use the solver branch

* Add an internal-miner feature and set up its dependencies

* Remove 'Experimental' from mining RPC docs

* Fix a nightly clippy::question_mark lint

* Move a byte array utility function to zebra-chain

* fixup! Add an internal-miner feature and set up its dependencies

* Add an equihash::Solution::solve() method with difficulty checks

* Check solution is valid before returning it

* Add a TODO to check for peers before mining

* Move config validation into GetBlockTemplateRpcImpl::new()

* fixup! fixup! Add an internal-miner feature and set up its dependencies

* Use the same generic constraints for GetBlockTemplateRpcImpl struct and impls

* Start adding an internal miner component

* Add the miner task to the start command

* Add basic miner code

* Split out a method to mine one block

* Spawn to a blocking thread

* Wait until a valid template is available

* Handle shutdown

* Run mining on low priority threads

* Ignore some invalid solutions

* Use a difference nonce for each solver thread

* Update TODOs

* Change the patch into a renamed dependency to simplify crate releases

* Clean up instrumentation and TODOs

* Make RPC instances cloneable and clean up generics

* Make LongPollId Copy so it's easier to use

* Add API to restart mining if there's a new block template

* Actually restart mining if there's a new block template

* Tidy instrumentation

* fixup! Move config validation into GetBlockTemplateRpcImpl::new()

* fixup! Make RPC instances cloneable and clean up generics

* Run the template generator and one miner concurrently

* Reduce logging

* Fix a bug in getblocktemplate RPC tip change detection

* Work around some watch channel change bugs

* Rate-limit template changes in the receiver

* Run one mining solver per available core

* Use updated C code with double-free protection

* Update to the latest solver branch

* Return and submit all valid solutions

* Document what INPUT_LENGTH means

* Fix watch channel change detection

* Don't return early when a mining task fails

* Spawn async miner tasks to avoid cooperative blocking, deadlocks, and improve shutdown responsiveness

* Make existing parallelism docs and configs consistent

* Add a mining parallelism config

* Use the minimum of the configured or available threads for mining

* Ignore optional feature fields in tests

* Downgrade some frequent logs to debug

* Document new zebrad features and tasks

* Describe the internal-miner feature in the CHANGELOG

* Update dependency to de-duplicate equihash solutions

* Use futures::StreamExt instead of TryStreamExt

* Fix a panic message typo
2024-01-11 14:41:01 +00:00
Arya 81a3a2aacf
add(docs): Note default path to config in docs (#8143)
* Note default path to config file in README and docs

* Applies suggestions from review & adds new test config file

* removes default path for 'Other' OSes from ZebradConfig docs

* fixes config_tests
2024-01-10 06:53:32 +00:00
dependabot[bot] f21d7c6934
build(deps): bump the prod group with 6 updates (#8125)
* build(deps): bump the prod group with 6 updates

Bumps the prod group with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [futures](https://github.com/rust-lang/futures-rs) | `0.3.29` | `0.3.30` |
| [tokio](https://github.com/tokio-rs/tokio) | `1.35.0` | `1.35.1` |
| [metrics](https://github.com/metrics-rs/metrics) | `0.21.1` | `0.22.0` |
| [metrics-exporter-prometheus](https://github.com/metrics-rs/metrics) | `0.12.2` | `0.13.0` |
| [reqwest](https://github.com/seanmonstar/reqwest) | `0.11.22` | `0.11.23` |
| [owo-colors](https://github.com/jam1garner/owo-colors) | `3.5.0` | `4.0.0` |


Updates `futures` from 0.3.29 to 0.3.30
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.29...0.3.30)

Updates `tokio` from 1.35.0 to 1.35.1
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.35.0...tokio-1.35.1)

Updates `metrics` from 0.21.1 to 0.22.0
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/compare/metrics-v0.21.1...metrics-v0.22.0)

Updates `metrics-exporter-prometheus` from 0.12.2 to 0.13.0
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/compare/metrics-exporter-prometheus-v0.12.2...metrics-exporter-prometheus-v0.13.0)

Updates `reqwest` from 0.11.22 to 0.11.23
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.22...v0.11.23)

Updates `owo-colors` from 3.5.0 to 4.0.0
- [Commits](https://github.com/jam1garner/owo-colors/compare/v3.5.0...v4.0.0)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: metrics
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: metrics-exporter-prometheus
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: owo-colors
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>

* update all metric macros

* fix deprecated function

* fix duplicated deps

* Fix an incorrect gauge method call

* Expand documentation and error messages for best chain length

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: teor <teor@riseup.net>
2024-01-02 01:26:54 +00:00
Alfredo Garcia d1e72c114f
docs(shielded-scan) Improve feature docs (#8114)
* improve shielded-scan feature docs

* apply suggestions from review

* Fix links and typos

* Use the right birthday height for ZECpages vk

---------

Co-authored-by: Marek <mail@marek.onl>
2023-12-21 17:05:56 +00:00
Alfredo Garcia 085bfdc067
feat(scanner): Don't scan and log if we are below sapling height (#8121)
* do not scan and notify if we are below sapling height

* separate tip logic check into its own loop
2023-12-21 13:44:27 +00:00
teor ad015e04d9
Move history tree and value balance to typed column families (#8115)
* impl TryFrom<zcash_primitives::BlockHeight> for Height

* Add type-safe read and write database methods

* Only allow typed access to the scanner DB

* Update docs

* Implement a common method as a trait

* Fix imports

* Tidy state imports

* Activate tracing logging macros in the whole scanner crate

* Fix dead code warnings

* Use a more sensible export order

* Remove a 1.72 lint exception now 1.74 is stable

* Switch history trees over to TypedColumnFamily, and remove redundant code

* Add typed batch creation methods, and switch history trees to them

* Convert ValueBalance to typed column families

* Make the APIs compatible after a merge

* Use `ZebraDb` instead of `DiskDb` where needed

---------

Co-authored-by: Marek <mail@marek.onl>
2023-12-20 23:20:57 +00:00
dependabot[bot] 3c8b93d986
build(deps): bump the prod group with 4 updates (#8117)
Bumps the prod group with 4 updates: [thiserror](https://github.com/dtolnay/thiserror), [sentry](https://github.com/getsentry/sentry-rust), [hyper](https://github.com/hyperium/hyper) and [metrics-exporter-prometheus](https://github.com/metrics-rs/metrics).


Updates `thiserror` from 1.0.50 to 1.0.51
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.50...1.0.51)

Updates `sentry` from 0.32.0 to 0.32.1
- [Release notes](https://github.com/getsentry/sentry-rust/releases)
- [Changelog](https://github.com/getsentry/sentry-rust/blob/master/CHANGELOG.md)
- [Commits](https://github.com/getsentry/sentry-rust/compare/0.32.0...0.32.1)

Updates `hyper` from 0.14.27 to 0.14.28
- [Release notes](https://github.com/hyperium/hyper/releases)
- [Changelog](https://github.com/hyperium/hyper/blob/v0.14.28/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.27...v0.14.28)

Updates `metrics-exporter-prometheus` from 0.12.1 to 0.12.2
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/commits)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: sentry
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: hyper
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: metrics-exporter-prometheus
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-18 22:03:41 +00:00
Alfredo Garcia 92758a0d9f
feat(scanner): Restart scanning where left (#8080)
* start scanner where it was left

* fix tests

* add a `scan_start_where_left` test

* refactor a log msg

* fix some comments

* remove function

* fix doc comment

* clippy

* fix `sapling_keys_and_last_scanned_heights()`

* simplify start height

* i went too far, revert some changes back

* change log info to every 10k blocks

* fix build

* Update height snapshot code and check last height is consistent

* Add strictly before and strictly after database key gets

* Move to the previous key using strictly before ops

* Assert that keys are only inserted once

* Update the index in each loop

* Update snapshots

* Remove debugging code

* start scanning at min available height

---------

Co-authored-by: teor <teor@riseup.net>
2023-12-13 20:16:26 +00:00
teor fa4c80f35a
Send all zebrad logs to the journal under systemd (#7965) 2023-12-12 13:15:55 +00:00
teor 3318eaaa22
test(scan): Add raw database format snapshots to the scanner (#8075)
* Make some scanner storage methods more flexible

* Move tests to a submodule and expose test functions and constants

* Make scanner functions clearer and easier to use

* Simplify state snapshot test code

* Add raw data snapshot tests for the scanner

* Add snapshots

* Fix import path

* Fix import conditional compilation

* fix imports

* fix imports 2

* Put read and write db exports together

* Remove confusing IntoDisk/FromDisk impl

* Fix an incorrect unused method that could panic

* Delete a test that is no longer valid

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2023-12-12 07:45:12 +00:00
Arya 5bdad1bcaa
Release Zebra 1.5.0 (#8027)
* adds a summary to changelog

* chore: Release

* Apply suggestions from code review

* updates Changelog.md

* updates changelog.md

* mentions/updates MSRV

* updates checkpoints

* updates ESTIMATE_RELEASE_HEIGHT

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* updates estimated release height

* combines changelog entries about scanner db

* zebra-scan is still in alpha

* Update CHANGELOG.md

* update release checklist for zebra-scan

* use fastmod to update zebra-scan version

* corrects zebra-scan version requirement in zebrad

* Applies suggestions from review

* updates changelog

* Update zebrad/src/components/sync/end_of_support.rs

---------

Co-authored-by: teor <teor@riseup.net>
2023-12-12 00:44:15 +00:00
Alfredo Garcia cdfbecf5f5
avoid starting the scan task if no keys to scan are present (#8059) 2023-12-06 01:56:43 +00:00
teor 4306a00f3c
Scan blocks with sapling keys and write the results to the database (#8040)
* Fix availability of tokio::time in scanner

* Create a function that parses a key into a list of keys

* Pass a ChainTipChange to the scanner function

* Convert a scanned block to a sapling result

* Make it easier to pass keys and blocks

* Increase scanner wait times

* Parse keys once at the start of the scan

* Get a block from the state instead of the tip

* Don't log secret keys, only log every 100,000 blocks

* Scan each block and add the results to storage

* Move blocking tasks into spawn_blocking()

* Update the acceptance test

* Use a dummy sapling tree size

* Use a larger dummy size
2023-12-03 21:58:48 +00:00
teor fc2576b27d
Standardise on the shielded-scan feature name (#8038) 2023-12-01 00:05:57 +00:00
teor db05845f98
change(scan): Use the on-disk database for keys and results (#8036)
* Expose IntoDisk and FromDisk in zebra-state

* Implement database serialization for SaplingScanningKey Strings

* Implement serialization for Vec<SaplingScannedResult> (Vec<transaction::Hash>)

* Implement seralization for SaplingScannedDatabaseIndex

* Add an is_empty() method

* Add a read method for a specific index, and document it

* Implement writing scanner results to the database

* Make read name more explicit

* Implement writing scanner keys

* Implement reading sapling keys

* Spawn blocking tasks correctly in async code

* Change storage results methods to use the database

* Update tests that use storage

* Use spawn_blocking() for database methods

* Change the check interval to slightly less than the block interval

* Expose raw database methods with shielded-scan

* fix `scan_task_starts` test

* minor doc change in test

---------

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2023-11-30 21:27:46 +00:00
teor 8c717c92dd
change(scan): Create a scanner storage database, but don't use it yet (#8031)
* Create an empty storage/db module

* Use ephemeral storage in tests

* Populate storage inside new() method

* Move scanner setup into an init() method

* Pass the network to scanner init

* Create a database but don't actually use it

* Skip shutdown format checks when skipping format upgrades

* Allow the scanner to skip launching format upgrades in production

* Refactor skipping format upgrades so it is consistent

* Allow checking configs for equality

* Restore Network import
2023-11-30 12:59:15 +00:00
teor cb9452c5e3
change(ci): When building crates individually, build all targets, and run clippy (#8024)
* Run clippy and build all targets on all crates individually

* Fix prod and test features for scanner deps

* Standardise dependency order

* Remove unnecessary async in tests

* Fix an unused import in a test

* Work around a no space left on device error

* Actually just use a larger runner
2023-11-30 00:51:20 +00:00
dependabot[bot] 9a57cec110
build(deps): bump the prod group with 2 updates (#8026)
Bumps the prod group with 2 updates: [clap](https://github.com/clap-rs/clap) and [sentry](https://github.com/getsentry/sentry-rust).


Updates `clap` from 4.4.9 to 4.4.10
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.4.9...v4.4.10)

Updates `sentry` from 0.31.8 to 0.32.0
- [Release notes](https://github.com/getsentry/sentry-rust/releases)
- [Changelog](https://github.com/getsentry/sentry-rust/blob/master/CHANGELOG.md)
- [Commits](https://github.com/getsentry/sentry-rust/compare/0.31.8...0.32.0)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: sentry
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-29 03:16:31 +00:00
teor 1d241afbaa
change(state): Expose ZebraDb methods that can create different kinds of databases (#8002)
* Provide access to DiskDb and DiskWriteBatch outside the state using a feature

* Actually let's export ZebraDb for the format upgrade code

* Pass column families to ZebraDb as an argument

* Allow the database kind to be changed in config.rs

* Use the state kind in finalized_state.rs

* Allow different database kinds in ZebraDb, but don't move the upgrade code yet

* Allow different database kinds in DiskDb

* Allow different database kinds in upgrade.rs, but don't split the upgrade code out yet

* Add new arguments to raw database tests

* Fix doc links

* Fix internal imports

* Fix unused code

* Update zebrad version metadata

* Create a specific state database delete function

* Fix state exports

* Fix zebrad tests

* Fix zebrad state write tests

* Make CI run again

* Fix dead code warnings for test methods

* Remove unnecessary async on some tests

* Fix logging required by tests

* Fix logging required in test itself

* Fix variable names

* Try to copy the message and add regexes
2023-11-28 13:49:11 +00:00
dependabot[bot] 35a7764b01
build(deps): bump the prod group with 3 updates (#8017)
Bumps the prod group with 3 updates: [clap](https://github.com/clap-rs/clap), [serde](https://github.com/serde-rs/serde) and [inferno](https://github.com/jonhoo/inferno).


Updates `clap` from 4.4.8 to 4.4.9
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.4.8...v4.4.9)

Updates `serde` from 1.0.192 to 1.0.193
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.192...v1.0.193)

Updates `inferno` from 0.11.18 to 0.11.19
- [Changelog](https://github.com/jonhoo/inferno/blob/main/CHANGELOG.md)
- [Commits](https://github.com/jonhoo/inferno/compare/v0.11.18...v0.11.19)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: inferno
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-28 03:29:47 +00:00
Alfredo Garcia 0f24c311a4
feat(zebra-scan): Connect with zebrad (#7989)
* connect zebrad with zebra-scan

* remove unwrap

* use tokio::sleep

* fix the task handler

* Don't panic on an empty state

---------

Co-authored-by: teor <teor@riseup.net>
2023-11-27 12:52:48 +00:00
Alfredo Garcia 681ae68455
build(deps): Bump what we can from the production group (#7996)
* bump what we can from the production group

* add duplicated dependencies
2023-11-27 03:26:29 +00:00
teor b5e16a6b26
fix(ci): Skip GCP CI jobs on PRs from external contributors, let mergify test them after approval (#7956)
* Skip external branches in ci-unit-tests-docker.yml

* Patch external repositories in ci-unit-tests-docker.patch.yml

* Revert ci-unit-tests-docker.patch.yml

* Add a separate external PR patch in ci-unit-tests-docker.patch-external.yml

* Document ci-unit-tests-docker.patch.yml

* Document ci-unit-tests-docker.yml

* Skip dependabot config validation for external PRs

* Trivial Rust code modification to make Rust workflows run

* Use correct method for checking source branch refs

* Fix the check condition

* Simplify job structure

* Use correct skip condition

* Make the start docs visible so we can link to them

* Skip integration tests on external PRs

* Allow external PRs to pass CD Docker tests

* Allow external PRs to pass Docs build and deploy
2023-11-22 19:43:29 +00:00
Arya 5e4c0f973f
security: Rate limit GetAddr responses (#7955)
* Updates ADDR_RESPONSE_LIMIT_DENOMINATOR to 4

* Moves logic getting a fraction of Zebra's peers to a method in the address book

* Adds and uses CachedPeerAddrs struct in inbound service

* moves and documents constant

* fixes test

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* updates docs

* renames sanitized_window method

* renames CachedPeerAddrs to CachedPeerAddrResponse

* updates test

* moves try_refresh to per request

* Make unused sanitization method pub(crate)

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* moves CachedPeerAddrResponse to a module

* updates unit test

* fixes unit test

* removes unnecessary condition

* clears cached getaddr response if it can't refresh for over a minute after the refresh time

* tests that inbound service gives out the same addresses for every Peers request before the refresh interval

* Applies suggestion from code review

* fixes doc link

* renames constant

* Fix docs on new constant

* applies suggestion from code review

* uses longer cache expiry time

* Adds code comments

---------

Co-authored-by: teor <teor@riseup.net>
2023-11-21 03:32:23 +00:00
Alfredo Garcia a22c8d5f42
feat(config): Allow to add keys to be scanned by the zebra-scan crate to config (#7949)
* allow user to add sapling keys to config

* apply code review suggestions

Co-authored-by: teor <teor@riseup.net>

---------

Co-authored-by: teor <teor@riseup.net>
2023-11-16 01:32:22 +00:00
dependabot[bot] 461c5aa680
build(deps): bump the prod group with 7 updates (#7938)
Bumps the prod group with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [clap](https://github.com/clap-rs/clap) | `4.4.7` | `4.4.8` |
| [serde](https://github.com/serde-rs/serde) | `1.0.190` | `1.0.192` |
| [toml](https://github.com/toml-rs/toml) | `0.8.6` | `0.8.8` |
| [tokio](https://github.com/tokio-rs/tokio) | `1.33.0` | `1.34.0` |
| [sentry](https://github.com/getsentry/sentry-rust) | `0.31.7` | `0.31.8` |
| [inferno](https://github.com/jonhoo/inferno) | `0.11.17` | `0.11.18` |
| [proptest](https://github.com/proptest-rs/proptest) | `1.3.1` | `1.4.0` |


Updates `clap` from 4.4.7 to 4.4.8
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.4.7...v4.4.8)

Updates `serde` from 1.0.190 to 1.0.192
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.190...v1.0.192)

Updates `toml` from 0.8.6 to 0.8.8
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.8.6...toml-v0.8.8)

Updates `tokio` from 1.33.0 to 1.34.0
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.33.0...tokio-1.34.0)

Updates `sentry` from 0.31.7 to 0.31.8
- [Release notes](https://github.com/getsentry/sentry-rust/releases)
- [Changelog](https://github.com/getsentry/sentry-rust/blob/master/CHANGELOG.md)
- [Commits](https://github.com/getsentry/sentry-rust/compare/0.31.7...0.31.8)

Updates `inferno` from 0.11.17 to 0.11.18
- [Changelog](https://github.com/jonhoo/inferno/blob/main/CHANGELOG.md)
- [Commits](https://github.com/jonhoo/inferno/compare/v0.11.17...v0.11.18)

Updates `proptest` from 1.3.1 to 1.4.0
- [Release notes](https://github.com/proptest-rs/proptest/releases)
- [Changelog](https://github.com/proptest-rs/proptest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/proptest-rs/proptest/compare/v1.3.1...v1.4.0)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: toml
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
- dependency-name: sentry
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: inferno
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: proptest
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-15 01:03:57 +00:00
teor af82a76d18
cleanup(clippy): Remove unnecessary `try_into()` with `cargo clippy --fix` (#7940)
* Remove unnecessary try_into() with cargo clippy --fix

* Manually remove some unnecessary try_from()s

* impl From<Diversifier> for pallas::Affine instead of TryFrom

* Remove unused imports

* cargo fmt --all

* Remove redundant comma
2023-11-14 20:31:49 +00:00
dependabot[bot] 2efc6bf4b2
build(deps): bump the prod group with 1 update (#7915)
Bumps the prod group with 1 update: [vergen](https://github.com/rustyhorde/vergen).

- [Release notes](https://github.com/rustyhorde/vergen/releases)
- [Commits](https://github.com/rustyhorde/vergen/compare/8.2.5...8.2.6)

---
updated-dependencies:
- dependency-name: vergen
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-07 19:30:58 +00:00
teor 477da1a774
Release Zebra 1.4.0 (#7911)
* Add a new known issue to the README

* Update the changelog

* Set release height around 8 November

* chore: Release
2023-11-06 12:51:41 +00:00
teor 628b3e39af
fix(net): Add outer timeouts for critical network operations to avoid hangs (#7869)
* Refactor out try_to_sync_once()

* Add outer timeouts for obtaining and extending tips

* Refactor out request_genesis_once()

* Wrap genesis download once in a timeout

* Increase the genesis timeout to avoid denial of service from old nodes

* Add an outer timeout to mempool crawls

* Add an outer timeout to mempool download/verify

* Remove threaded mutex blocking from the inbound service

* Explain why inbound readiness never hangs

* Fix whitespace that cargo fmt doesn't

* Avoid hangs by always resetting the past lookahead limit flag

* Document block-specific and syncer-wide errors

* Update zebrad/src/components/sync.rs

Co-authored-by: Marek <mail@marek.onl>

* Use correct condition for log messages

Co-authored-by: Marek <mail@marek.onl>

* Keep lookahead reset metric

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
Co-authored-by: Marek <mail@marek.onl>
2023-11-02 15:00:18 +00:00
teor afbe807060
change(docs): Replace doc.zebra.zfnd.org API documentation with docs.rs (#7886)
* Simple replacements of doc.zebra.zfnd.org with docs.rs

* Manual fixes for specific main/internal/external docs

* Point developer docs to doc-internal.zebra.zfnd.org

* fastmod --glob '\!.git' -- doc.zebra.zfnd.org/zebrad docs.rs/zebrad/latest/zebrad

* Manually remove any remaining doc.zfnd.zebra.org links

* Remove the external docs job

* Add changelog entry and fix links

* Fix links that were broken before this PR
2023-11-01 22:09:40 +00:00
dependabot[bot] bd062941b9
build(deps): bump the prod group with 2 updates (#7881)
Bumps the prod group with 2 updates: [clap](https://github.com/clap-rs/clap) and [serde_json](https://github.com/serde-rs/json).


Updates `clap` from 4.4.6 to 4.4.7
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.4.6...v4.4.7)

Updates `serde_json` from 1.0.107 to 1.0.108
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.107...v1.0.108)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-31 13:15:47 +00:00
dependabot[bot] b0555c0e61
build(deps): bump the test group with 1 update (#7876)
* build(deps): bump the test group with 1 update

Bumps the test group with 1 update: [tempfile](https://github.com/Stebalien/tempfile).

- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/commits)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: test
...

Signed-off-by: dependabot[bot] <support@github.com>

* add duplicated dependency

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2023-10-30 22:08:34 +00:00
Alfredo Garcia f3048653c8
chore: fix typos (#7877)
Co-authored-by: Dimitris Apostolou <dimitris.apostolou@icloud.com>
2023-10-30 20:06:35 +00:00
dependabot[bot] d0e5b2f3ef
build(deps): bump the async group with 2 updates (#7872)
Bumps the async group with 2 updates: [futures](https://github.com/rust-lang/futures-rs) and [tokio-util](https://github.com/tokio-rs/tokio).


Updates `futures` from 0.3.28 to 0.3.29
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.28...0.3.29)

Updates `tokio-util` from 0.7.9 to 0.7.10
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/commits)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: async
- dependency-name: tokio-util
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: async
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-30 18:26:53 +00:00
dependabot[bot] ad0051c228
build(deps): bump the formats group with 1 update (#7874)
Bumps the formats group with 1 update: [serde](https://github.com/serde-rs/serde).

- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.189...v1.0.190)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: formats
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-30 18:26:30 +00:00
Arya 0a3790b73e
change(consensus): Remove Sprout and Sapling parameter download task and debug_skip_preload config (#7844)
* removes groth16 download task

* updates docs.

* Adds new config to test configs

* fixes compatibility with past configs

* uses inner config to deserialize removed field for compatibility

* update docs

* updates latest config

* Applies suggestions from code review

* Avoid duplicating hard-coded default values

---------

Co-authored-by: teor <teor@riseup.net>
2023-10-27 06:12:57 +00:00
teor 920ee14512
fix(doc): Fix broken links to `zebra_network` and `zebra_state` `Config` structs on doc.zebra.zfnd.org (#7838)
* Use full module paths to avoid a rustdoc bug

* Exclude zebra-test from external docs
2023-10-27 01:05:26 +00:00
teor 71a9865b6c
remove(cmd): Remove `zebrad download` command, because it no longer does anything (#7819)
* Remove the `zebrad download` command

* Reorder command.rs imports

* Remove zcash-params/Dockerfile and the main Dockerfile commands that use it

* Stop building zcash-params Docker images in CI

* Update CHANGELOG for `zebrad download` removal

* Clarify why the image is smaller

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2023-10-25 23:42:27 +00:00
teor aad883b68a
change(consensus): Build Sprout and Sapling parameters into the zebrad binary, so a download server isn't needed (#7800)
* Build sapling parameters into zebrad

* Load the sapling parameters from wagyu-zcash-parameters

* Reword logs

* wget https://github.com/zcash/zcash/raw/master/src/rust/src/sprout-groth16.vk

* Load sprout from binary and remove download code

* Remove download examples binaries

* Remove unused features and dependencies

* Remove docs and comments about downloading parameters

* Add CHANGELOG entry

* Remove parameter cache steps in workflows

* Update comment about downloading

Co-authored-by: Marek <mail@marek.onl>

* Fix outdated timeout comments and short timeouts

---------

Co-authored-by: Marek <mail@marek.onl>
2023-10-24 23:27:24 +00:00
dependabot[bot] e56b66d182
build(deps): bump the app group with 1 update (#7802)
Bumps the app group with 1 update: [toml](https://github.com/toml-rs/toml).

- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.8.2...toml-v0.8.3)

---
updated-dependencies:
- dependency-name: toml
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: app
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-23 19:18:56 +00:00
teor 64f777274c
fix(security): fix concurrency issues in tree key formats, and CPU usage in genesis tree roots (#7392)
* Add tree key format and cached root upgrades

* Document the changes in the upgrades

* Remove unnecessary clippy::unwrap_in_result

* Fix database type

* Bump state version

* Skip some checks if the database is empty

* Fix tests for a short state upgrade

* Disable format checks in some tests

* Document state performance issues

* Clarify upgrade behaviour

* Clarify panic messages

* Delete incorrect genesis trees write code

* Fix metrics handling for genesis

* Remove an unused import

* Explain why genesis anchors are ok

* Update snapshots

* Debug a failing test

* Fix some tests

* Fix missing imports

* Move the state check in a test

* Fix comment and doc typos

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>

* Clarify what a long upgrade is

* Rename unused function arguments

Co-authored-by: Marek <mail@marek.onl>

* Add all_unordered log regex matching methods

* Fix timing issues with version upgrades and other logs

* Fix argument name in docs

Co-authored-by: Marek <mail@marek.onl>

* Explain match until first for all regexes behaviour better

---------

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
2023-10-19 14:50:46 +00:00
teor 08ce2ad500
feature(mining): Enable mining RPCs by default in production builds (#7740)
* Enable getblocktemplate-rpcs in production builds

* Update mining instructions

* Consistently use ZF funding stream addresses in examples

* Simplify skip_getblocktemplate()

* Set the test miner address based on the network

* Correctly skip checks in lightwalletd send transactions test

* Wait longer for transaction query

* Wait longer before re-checking the mempool

* Skip mempool checks if it wasn't queried by lightwalletd

* rustfmt

* clippy and rustfmt

* Fix attribute issue

* Fix typo

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-10-18 04:15:17 +00:00
teor 2a077f21c7
fix(ci): Skip mempool checks if it wasn't queried by lightwalletd (#7744)
* Correctly skip checks in lightwalletd send transactions test

* Wait longer for transaction query

* Wait longer before re-checking the mempool

* Skip mempool checks if it wasn't queried by lightwalletd

* rustfmt

* clippy and rustfmt

* Fix attribute issue
2023-10-17 02:56:00 +00:00
dependabot[bot] 5b9877a520
build(deps): bump the formats group with 1 update (#7746)
Bumps the formats group with 1 update: [regex](https://github.com/rust-lang/regex).

- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.10.0...1.10.2)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: formats
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-17 00:25:55 +00:00