Commit Graph

260 Commits

Author SHA1 Message Date
Alfredo Garcia f3048653c8
chore: fix typos (#7877)
Co-authored-by: Dimitris Apostolou <dimitris.apostolou@icloud.com>
2023-10-30 20:06:35 +00:00
Arya 0a3790b73e
change(consensus): Remove Sprout and Sapling parameter download task and debug_skip_preload config (#7844)
* removes groth16 download task

* updates docs.

* Adds new config to test configs

* fixes compatibility with past configs

* uses inner config to deserialize removed field for compatibility

* update docs

* updates latest config

* Applies suggestions from code review

* Avoid duplicating hard-coded default values

---------

Co-authored-by: teor <teor@riseup.net>
2023-10-27 06:12:57 +00:00
teor 64f777274c
fix(security): fix concurrency issues in tree key formats, and CPU usage in genesis tree roots (#7392)
* Add tree key format and cached root upgrades

* Document the changes in the upgrades

* Remove unnecessary clippy::unwrap_in_result

* Fix database type

* Bump state version

* Skip some checks if the database is empty

* Fix tests for a short state upgrade

* Disable format checks in some tests

* Document state performance issues

* Clarify upgrade behaviour

* Clarify panic messages

* Delete incorrect genesis trees write code

* Fix metrics handling for genesis

* Remove an unused import

* Explain why genesis anchors are ok

* Update snapshots

* Debug a failing test

* Fix some tests

* Fix missing imports

* Move the state check in a test

* Fix comment and doc typos

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>

* Clarify what a long upgrade is

* Rename unused function arguments

Co-authored-by: Marek <mail@marek.onl>

* Add all_unordered log regex matching methods

* Fix timing issues with version upgrades and other logs

* Fix argument name in docs

Co-authored-by: Marek <mail@marek.onl>

* Explain match until first for all regexes behaviour better

---------

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Arya <aryasolhi@gmail.com>
2023-10-19 14:50:46 +00:00
teor 08ce2ad500
feature(mining): Enable mining RPCs by default in production builds (#7740)
* Enable getblocktemplate-rpcs in production builds

* Update mining instructions

* Consistently use ZF funding stream addresses in examples

* Simplify skip_getblocktemplate()

* Set the test miner address based on the network

* Correctly skip checks in lightwalletd send transactions test

* Wait longer for transaction query

* Wait longer before re-checking the mempool

* Skip mempool checks if it wasn't queried by lightwalletd

* rustfmt

* clippy and rustfmt

* Fix attribute issue

* Fix typo

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-10-18 04:15:17 +00:00
teor 2a077f21c7
fix(ci): Skip mempool checks if it wasn't queried by lightwalletd (#7744)
* Correctly skip checks in lightwalletd send transactions test

* Wait longer for transaction query

* Wait longer before re-checking the mempool

* Skip mempool checks if it wasn't queried by lightwalletd

* rustfmt

* clippy and rustfmt

* Fix attribute issue
2023-10-17 02:56:00 +00:00
teor ae52e3d23d
change(ui): Enable the progress bar feature by default, but only show progress bars when the config is enabled (#7615)
* Add a progress bar config that is disabled unless the feature is on

* Simplify the default config

* Enable the progress bar feature by default, but require the config

* Rename progress bars config to avoid merge conflicts

* Use a log file when the progress bar is activated

* Document how to configure progress bars

* Handle log files in config_tests and check config path

* Fix doc link

* Fix path check

* Fix config log matching

* Fix clippy warning

* Add tracing to config tests

* It's zebrad not zebra

* cargo fmt --all

* Update release for config file changes

* Fix config test failures

* Allow printing to stdout in a method
2023-10-12 00:25:37 +00:00
Arya 931e0a7587
fix(tests): Checking if transactions get into the mempool fails in lightwalletd tests (#7644)
* copy zcash/lightwalletd service.proto

* uses correct assert & minor cleanup

* adds comment about checking service.proto file when

* adds workaround in send_transaction_test

* Update docs

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: teor <teor@riseup.net>
2023-10-10 02:00:51 +00:00
teor 0faa0697d5
change(state): Only do continuous format checks in CI (#7627)
* Only do continuous format checks if enabled by a config

* Run continuous format checks in CI
2023-09-27 21:42:43 +00:00
Alfredo Garcia 09c1f994f0
remove intermittent asserts in test (#7600) 2023-09-21 03:20:55 +00:00
Alfredo Garcia 80da28b3e3
tests(rpc): Add fixed test vectors for `z_getsubtreesbyindex` from zcashd to zebra (#7515)
* add fixed test vectors for `z_getsubtreesbyindex`

* change to snapshots

* add test to docker

* remove assert lines from snapshots

* add more tests

* change test description

* change test name

* run both tests together

* wait for state version update in test

* Run one test at a time to avoid state locking issues

---------

Co-authored-by: teor <teor@riseup.net>
2023-09-20 23:40:21 +00:00
teor 7a7d79dfaf
fix(state): Use correct end heights for end of block subtrees during the full sync (#7566)
* Avoid manual handling of previous sapling trees by using iterator windows instead

* Avoid manual sapling subtree index handling by comparing prev and current subtree indexes instead

* Simplify adding notes by using the exact number of remaining notes

* Simplify by skipping the first block, because it can't complete a subtree

* Re-use existing tree update code

* Apply the sapling changes to orchard subtree updates

* add a reverse database column family iterator function

* Make skipping the lowest tree independent of iteration order

* Move new subtree checks into the iterator, rename to end_height

* Split subtree calculation into a new method

* Split the calculate and write methods

* Quickly check the first subtree before running the full upgrade

* Do the quick checks every time Zebra runs, and refactor slow check error handling

* Do quick checks for orchard as well

* Make orchard tree upgrade match sapling upgrade code

* Upgrade subtrees in reverse height order

* Bump the database patch version so the upgrade runs again

* Reset previous subtree upgrade data before doing this one

* Add extra checks to subtree calculation to diagnose errors

* Use correct heights for subtrees completed at the end of a block

* Add even more checks to diagnose issues

* Instrument upgrade methods to improve diagnostics

* Prevent modification of re-used trees

* Debug with subtree positions as well

* Fix an off-by-one error with completed subtrees

* Fix typos and confusing comments

Co-authored-by: Marek <mail@marek.onl>

* Fix mistaken previous tree handling and end tree comments

* Remove unnecessary subtraction in remaining leaves calc

* Log heights when assertions fail

* Fix new subtree detection filter

* Move new subtree check into a method, cleanup unused code

* Remove redundant assertions

* Wait for subtree upgrade before testing RPCs

* Fix subtree search in quick check

* Temporarily upgrade subtrees in forward height order

* Clarify some comments

* Fix missing test imports

* Fix subtree logging

* Add a comment about a potential hang with future upgrades

* Fix zebrad var ownership

* Log more info when add_subtrees.rs fails

* cargo fmt --all

* Fix unrelated clippy::unnecessary_unwrap

* cargo clippy --fix --all-features --all-targets; cargo fmt --all

* Stop the quick check depending on tree de-duplication

* Refactor waiting for the upgrade into functions

* Wait for state upgrades whenever the cached state is updated

* Wait for the testnet upgrade in the right place

* Fix unused variable

* Fix a subtree detection bug and comments

* Remove an early reference to reverse direction

* Stop skipping subtrees completed at the end of blocks

* Actually fix new subtree code

---------

Co-authored-by: Marek <mail@marek.onl>
2023-09-19 14:49:36 +00:00
Alfredo Garcia 91ec8b910b
change(tests): update tests for new lightwalletd version used in Zebra (#7349)
* update docs for new lightwalletd version used in Zebra

* update some asserts

* add `RUST_LOG` to docker `entrypoint.sh` file

* clarify zecwallet-cli is untested with zcash/lightwalletd fork in the book

* revert entrypoint rust log changes

* remove sleep

* revert doc changes

* change formatting in expected output

* try sleep now that CI is working

* try sending more transactions

* remove tests with failures

* remove newline
2023-09-14 05:25:42 +00:00
Alfredo Garcia a63fe225a0
change(tests): Obtain lightwalletd tip from logs instead of grpc (#7507)
* obtain lightwalletd from the logs

* revert #7332

* temporarily stop checking that the update sync has a good cached state

* Revert "temporarily stop checking that the update sync has a good cached state"

This reverts commit 0fb10938d9.

* add `save_to_disk` to workflow

* Fix lightwalletd height_grep_text for ECC fork

* Add a TODO about blocking the async executor

---------

Co-authored-by: teor <teor@riseup.net>
2023-09-13 03:33:09 +00:00
Marek dc6aa708d0
Test `z_getsubtreesbyindex` using a lightwalletd gRPC request (#7521)
* Add lightwalletd's protobuf types

* Don't explicitly derive `Eq` for enums

I got bitten by this bug: https://github.com/tokio-rs/prost/issues/332
when I added the enum `ShieldedProtocol` to the file `service.proto`.
The problem is that `prost` implicitly derives `Eq` for enums, so
deriving it explicitly via `type_attribute` causes a conflict. Lukily,
there is another method `message_attribute` that operates only on
messages and not enums.

* Test the `z_getsubtreesbyindex` RPC

* Fix a typo

* Add test vectors
2023-09-12 23:59:56 +00:00
teor 29d8e90ccf
fix(ci): Increase test launch and shutdown times to fix CI failures (#7499)
* Increase launch times to help fix failures in rpc_conflict and non_blocking_logger tests

* Add extra task spawn and shutdown logs to start.rs
2023-09-07 00:04:02 +00:00
Alfredo Garcia 38caaafc02
change(docs): update docs for new lightwalletd version used in Zebra (#7427)
* upgrade docs for new lightwalletd repo zebra is using now

* Update book/src/user/lightwalletd.md

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2023-08-30 22:30:20 +00:00
Arya eec467a173
should fix the timeouts (#7332) 2023-08-15 22:16:09 +00:00
Alfredo Garcia b322748b66
add(rpc): note tree sizes to `getblock` api (#7278)
* add the basics

* add some docs, move code

* upgrade compact formats to https://github.com/zcash/lightwalletd/blob/v0.4.15/walletrpc/compact_formats.proto

* add a test for in sync chain

* test changing to ecc lightwalletd

* revert change of lightwalletd repo (already merged to main)

* add debug log to see whats going on with the test

* change log to tracing::info

* remove log line

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-08-15 18:48:50 +00:00
Arya 84927d3dd1
Updates test expected logs (#7315)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-08-11 03:53:31 +00:00
teor c3e7d749ed
fix(log): Limit verbose RPC responses in logs (#7258)
* Truncate verbose RPC responses in test logs

* Downgrade verbose block template state log to debug
2023-07-21 20:35:56 +00:00
teor be5cfad07f
change(state): Prepare for in-place database format upgrades, but don't make any format changes yet (#7031)
* Move format upgrades to their own module and enum

* Launch a format change thread if needed, and shut it down during shutdown

* Add some TODOs and remove a redundant timer

* Regularly check for panics in the state upgrade task

* Only run example upgrade once, change version field names

* Increment database format to 25.0.2: add format change task

* Log the running and initial disk database format versions on startup

* Add initial disk and running state versions to cached state images in CI

* Fix missing imports

* Fix typo in logs workflow command

* Add a force_save_to_disk argument to the CI workflow

* Move use_internet_connection into zebrad_config()

* fastmod can_spawn_zebrad_for_rpc can_spawn_zebrad_for_test_type zebra*

* Add a spawn_zebrad_without_rpc() function

* Remove unused copy_state() test code

* Assert that upgrades and downgrades happen with the correct versions

* Add a kill_and_return_output() method for tests

* Add a test for new_state_format() versions (no upgrades or downgrades)

* Add use_internet_connection to can_spawn_zebrad_for_test_type()

* Fix workflow parameter passing

* Check that reopening a new database doesn't upgrade (or downgrade) the format

* Allow ephemeral to be set to false even if we don't have a cached state

* Add a test type that will accept any kind of state

* When re-using a directory, configure the state test config with that path

* Actually mark newly created databases with their format versions

* Wait for the state to be opened before testing the format

* Run state format tests on mainnet and testnet configs (no network access)

* run multiple reopens in tests

* Test upgrades run correctly

* Test that version downgrades work as expected (best effort)

* Add a TODO for testing partial updates

* Fix missing test arguments

* clippy if chain

* Fix typo

* another typo

* Pass a database instance to the format upgrade task

* Fix a timing issue in the tests

* Fix version matching in CI

* Use correct env var reference

* Use correct github env file

* Wait for the database to be written before killing Zebra

* Use correct workflow syntax

* Version changes aren't always upgrades

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-07-13 21:36:15 +00:00
teor f6afec2be8
fix(ci): Increase peer cache startup wait time and test time (#7169)
* Increase peer cache test time

* Wait longer for DNS responses before writing peers to disk
2023-07-07 03:35:41 +00:00
teor 147b8fa3a8
cleanup(rust): Fix new nightly clippy warnings (#7135)
* Fix "comparison is always true" warning

* Add missing Send bound

* cargo clippy --fix --all-features --all-targets

* incorrect implementation of clone on a Copy type

* cargo fmt --all
2023-07-05 07:11:27 +00:00
Arya b40fc9b032
change(network): Configurable maximum connections per IP (#7013)
* Adds config field

* adds new generated config

* Lint

* fixes config_tests
2023-06-20 05:11:45 +00:00
Alfredo Garcia 219d472270
fix(compatibility): Replace or add RPC content type header when applicable (#6885)
* ignore client supplied content-type header and use json always

* rename method

* add one more check to test

* Add missing proptest-impl dependency from zebrad to zebra-rpc

* change to replace only specific content type

* remove cargo mods

* refactor `insert_or_replace_content_type_header`

* add security comments

* allow variants of text/plain ocntent_type

---------

Co-authored-by: teor <teor@riseup.net>
2023-06-14 19:02:55 +00:00
teor 38206373b4
fix(ci): Add jobs to allow the full sync to finish before the first stable release (#6846)
* Delete an unused CI job that was previously partially deleted

* Add 2 more jobs to the full sync test

* Increase Rust test time: current expected time is 60 hours
2023-06-07 15:35:32 +00:00
Arya 59086c7d00
fix(zebrad): accept default subcommand arguments and print consistent usage information for top-level 'help' subcommand (#6801)
* updates Cargo.toml

* Migrate to abscissa 0.7.0

* Avoid panic from calling color_eyre::install twice

* Uses 'start' as the default subcommand

* updates default cmd logic

* Fixes minor cli issues

* removes outdated check in acceptance test

* Adds a test for process_cli_args, fixes version_args test.

Adds -V to process_cli_args match case

* Revert "fix(clippy): Silence future-incompat warnings until we upgrade Abscissa (#6024)"

This reverts commit dd90f79b48.

* Drops the worker guard to flush logs when zebra shuts down

* Adds cargo feature to clap

* restores process_cli_args

* updates deny.toml

* Updates EntryPoint help template

* Updates subcommand help msgs

* removes trailing whitespace, capitalizes sentences

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* revert parts of revert "Revert fix(clippy): Silence future-incompat warnings until we upgrade Abscissa"

* Applies suggestions from code review

* Moves EntryPoint to its own module

* fixes version_args test

* Updates changelog

* Prunes redundant test cases

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Revert "Prunes redundant test cases"

This reverts commit 3f73979184.

* Update zebrad/src/commands/entry_point.rs

Co-authored-by: teor <teor@riseup.net>

* Add missing import

* Updates `process_cli_args` to return a result

---------

Co-authored-by: teor <teor@riseup.net>
2023-06-07 06:03:42 +00:00
teor 04e96c2526
feat(net): Cache a list of useful peers on disk (#6739)
* Rewrite some state cache docs to clarify

* Add a zebra_network::Config.cache_dir for peer address caches

* Add new config test files and fix config test failure message

* Create some zebra-chain and zebra-network convenience functions

* Add methods for reading and writing the peer address cache

* Add cached disk peers to the initial peers list

* Add metrics and logging for loading and storing the peer cache

* Replace log of useless redacted peer IP addresses

* Limit the peer cache minimum and maximum size, don't write empty caches

* Add a cacheable_peers() method to the address book

* Add a peer disk cache updater task to the peer set tasks

* Document that the peer cache is shared by multiple instances unless configured otherwise

* Disable peer cache read/write in disconnected tests

* Make initial peer cache updater sleep shorter for tests

* Add unit tests for reading and writing the peer cache

* Update the task list in the start command docs

* Modify the existing persistent acceptance test to check for peer caches

* Update the peer cache directory when writing test configs

* Add a CacheDir type so the default config can be enabled, but tests can disable it

* Update tests to use the CacheDir config type

* Rename some CacheDir internals

* Add config file test cases for each kind of CacheDir config

* Panic if the config contains invalid socket addresses, rather than continuing

* Add a network directory to state cache directory contents tests

* Add new network.cache_dir config to the config parsing tests
2023-06-06 08:28:14 +00:00
teor 628ddd64ea
fix(test): Skip editor files and other hidden files in test configs directory (#6796)
* Skip editor files and other hidden files in test configs directory

* Also ignore files starting with `#`
2023-06-05 23:56:08 +00:00
Alfredo Garcia eb07bb31d6
rename(state): Rename state verifiers and related code (#6762)
* rename verifiers

* rename `PreparedBlock` to `SemanticallyVerifiedBlock`

* rename `CommitBlock` to `SemanticallyVerifiedBlock`

* rename `FinalizedBlock` to `CheckpointVerifiedBlock`

* rename `CommitFinalizedBlock` to `CommitCheckpointVerifiedBlock`

* rename `FinalizedWithTrees` to `ContextuallyVerifiedBlockWithTrees`

* rename `ContextuallyValidBlock` to `ContextuallyVerifiedBlock`

* change some `finalized` variables or function arguments to `checkpoint_verified`

* fix docs

* document the difference between `CheckpointVerifiedBlock` and `ContextuallyVerifiedBlock`

* fix doc links

* apply suggestions to request

Co-authored-by: Marek <mail@marek.onl>

* apply suggestions to service

Co-authored-by: Marek <mail@marek.onl>

* apply suggestions to finalized_state.rs and write.rs

Co-authored-by: Marek <mail@marek.onl>

* fmt

* change some more variable names

* change a few missing generics

* fix checkpoint log issue

* rename more `prepared` vars `semantically_verified`

* fix test regex

* fix test regex 2

---------

Co-authored-by: Marek <mail@marek.onl>
2023-06-01 12:29:03 +00:00
teor f9b5eb3a15
fix(log): Limit RPC failure log length, add details to RPC failure logs (#6754)
* Limit the RPC failure log length

* Add more specific logs for some RPC tests

* Make sure the block hash is always available for logging
2023-05-25 18:28:25 +00:00
teor 889e7c668e
fix(ci): Reduce number of sent transactions in tests (#6736)
* Reduce sent transactions in tests

* rustfmt
2023-05-22 12:05:40 +00:00
teor fadddae610
Check for early exits in the end of support task test (#6715) 2023-05-18 21:25:23 +00:00
Alfredo Garcia 7c67512cd5
feat(zebra-network): add user agent argument (#6601)
* add user agent as argument, use git to auto build zebra user agent

* try to fix test

* fix typo

* change expect text

* remove newline

* fix some docs

Co-authored-by: Marek <mail@marek.onl>

---------

Co-authored-by: Marek <mail@marek.onl>
2023-05-05 00:29:14 +00:00
Alfredo Garcia 58bd898f5b
feat(zebrad): Refuse to run zebrad when release is too old (#6351)
* refuse to run Zebra if it is too old

* update the release checklist to consider the constants

* bring newline back

* apply new end of support code

* attempt to add tests (not working yet)

* move eos to progress task

* move tests

* add acceptance test (not working)

* fix tests

* change to block height checks (ugly code)

* change warn days

* refactor estimated blocks per day, etc

* move end of support code to its own task

* change test

* fix some docs

* move constants

* remove uneeded conversions

* downgrade tracing

* reduce end of support time, fix ci changing debugs to info again

* update instructions

* add failure messages

* cargo lock update

* unify releaase name constant

* change info msg

* clippy fixes

* add a block explorer

* ignore testnet in end of support task

* change panic to 16 weeks

* add some documentation about end of support

* Tweak docs wording

---------

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-04-28 14:13:21 +00:00
teor 1461c912f9
change(ci): Generate mainnet checkpoints in CI (#6550)
* Add extra test type modes to support zebra-checkpoints

* Add Mainnet and Testnet zebra-checkpoints test harnesses

* Add zebra-checkpoints to test docker images

* Add zebra-checkpoints test entrypoints

* Add Mainnet CI workflow for zebra-checkpoints

* Enable zebra-checkpoints feature in the test image

* Use the same features for (almost) all the docker tests

* Make workflow features match Docker features

* Add a feature note

* Add a zebra-checkpoints test feature to zebrad

* Remove the "no cached state" testnet code

* Log a startup message to standard error when launching zebra-checkpoints

* Rename tests to avoid partial name conflicts

* Fix log formatting

* Add sentry feature to experimental docker image build

* Explain what ENTRYPOINT_FEATURES is used for

* Use the correct zebra-checkpoints path

* Silence zebrad logs while generating checkpoints

* Fix zebra-checkpoints log handling

* Re-enable waiting for zebrad to fully sync

* Add documentation for how to run these tests individually

* Start generating checkpoints from the last compiled-in checkpoint

* Fix clippy lints

* Revert changes to TestType

* Wait for all the checkpoints before finishing

* Add more stderr debugging to zebra-checkpoints

* Fix an outdated module comment

* Add a workaround for zebra-checkpoints launch/run issues

* Use temp dir and log what it is

* Log extra metadata about the zebra-checkpoints binary

* Add note about unstable feature -Z bindeps

* Temporarily make the test run faster and with debug info

* Log the original test command name when showing stdout and stderr

* Try zebra-checkpoints in the system path first, then the cargo path

* Fix slow thread close bug in dual process test harness

* If the logs are shown, don't say they are hidden

* Run `zebra-checkpoints --help` to work out what's going on in CI

* Build `zebra-utils` binaries for `zebrad` integration tests

* Revert temporary debugging changes

* Revert changes that were moved to another PR
2023-04-27 04:39:43 +00:00
teor d3ce022ecc
change(utils): Add a direct connection mode to zebra-checkpoints (#6516)
* Rename variables so it's clearer what they do

* Fully document zebra-checkpoints arguments

* Remove some outdated references to zcashd

* Add a json-conversion feature for converting JSON to valid Heights

* Simplify zebra-checkpoints code using conversion methods

* Track the last checkpoint height rather than the height gap

* Move all the CLI-specific code into a single function

* Remove cfg(feature) from the test RPC client API

* Move the RpcRequestClient into zebra-node-services

* Fix the case of RpcRequestClient

* Add transport and addr arguments to zebra-checkpoints

* Make zebra-checkpoints support both CLI and direct JSON-RPC connections

* Fix RpcRequestClient compilation

* Add a suggestion for zcashd authentication failures

* Document required features

* Handle differences in CLI & direct parameter and response formats

* Replace a custom function with an existing dependency function

* Add a checkpoint list test for minimum height gaps
2023-04-26 23:35:53 +00:00
teor 9c15b14f86
fix(ci): Avoid inbound service overloads in tests (#6537)
* Silence an extremely verbose error in zebra-consensus tests

This disables around 10,000 logs like:
2023-04-18T02:46:28.441662Z  WARN init{config=Config { checkpoint_sync: true, debug_skip_parameter_preload: false } network=Mainnet debug_skip_parameter_preload=true}: unexpected error: Closed in state request while verifying previous state checkpoints

* Increase the outbound connection interval to 100ms

* Start the inbound service as soon as possible, and the syncer last

* Increase acceptance test time limits to get more debug info

* Add more debug info to inbound service overload tracing messages
2023-04-23 13:41:38 +00:00
teor ec43d63ed2
change(log): Log a cute message for blocks that were mined by Zebra (off by default) (#6098)
* Mark Zebra coinbase transactions with extra coinbase data

* Log when we commit a block mined by Zebra to our state

* Reduce logging instrumentation during block writes

* Remove debug types in Zebra block log

* Add network and commit to write task logs

* Apply an allow-list before we log arbitrary user messages from blocks

* Rate-limit Zebra mined block logging to once every 1000 blocks

* Add mining configs for extra coinbase data and imitating zcashd, but don't use them yet

* Check CoinbaseData size limit when building transparent transactions

* Replace LIKE_ZCASHD constants with a config

* Take extra coinbase data from the configured string

* Update the zebrad configs in the tests with new config fields
2023-02-23 00:10:11 +00:00
teor 4daedbc789
fix(consensus): Check that Zebra's state contains the social consensus chain on startup (#6163)
* Allow missing docs directly on derived error types

* Make Request::BestChainBlockHash redirect to the ReadStateService

* Re-write the checkpoint_sync documentation based on the latest consensus rules

* Expose the underlying iterator for CheckpointList

* Validate existing state block hashes at startup, but ignore the result

* Monitor state block hash checkpoint task in the start command

* Fix indentation

* Make logging consistent

* Explain the config needed for full security

* Tidy required checkpoints docs, expand other docs

* Add security and deprecation changelog entries

* Replace task handle vector with a struct

* Add a test that this consensus-critical code actually runs and finishes

* Make some state methods and types available in tests

* Add missing docs

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-02-21 05:30:29 +00:00
teor ae21e36018
fix(rpc): Add `getblock` RPC fields to support the latest version of `zcash/lightwalletd` (#6134)
* Stabilise the BestChainBlockHash state request

* Always include the block hash in the getblock RPC response

* Make the lightwalletd integration tests compatible with zcash/lightwalletd

* Update getblock RPC snapshots

* Return the correct missing block error code
2023-02-14 08:52:58 +00:00
teor 311daaa19e
Add 2 full sync CI jobs and 10 hours to the Rust test time limit (#6103) 2023-02-06 21:16:28 +00:00
Arya d67b3b641d
removes excessive logs, adds docs. (#6102) 2023-02-05 22:18:18 +00:00
Arya 0aab3df731
fix(test): avoid failing getblocktemplate acceptance test when the chain tip changes (#6091)
* adds basic usage of long polling in gbt test

* adds !submit_old check before cancelling proposing a new template

* Removes break statement in long polling task

* Update zebrad/tests/common/rpc_client.rs

Co-authored-by: teor <teor@riseup.net>

* use blocking_send and watch channel

* fix "cannot block the current thread from within a runtime"

* Reduces interval between proposals and increases num proposals required.

* Runs rate-limiting sleeps in parallel to validation

* corrects comment.

---------

Co-authored-by: teor <teor@riseup.net>
2023-02-03 05:48:49 +00:00
teor 14fdd3002b
change(ci): Run block proposal tests in CI (#5963)
* Revert "Temporarily fix the code for a disabled test"

This reverts commit d915624417cc946e53aac76449e0b5b719e03d2a.

* Check every valid time source in the proposal tests

* Activate block proposal tests in CI

* Repeats block proposal test a few times at an interval to sample different mempool contents

* Increase the number of templates tested to 10

Co-authored-by: Arya <aryasolhi@gmail.com>

---------

Co-authored-by: arya2 <aryasolhi@gmail.com>
2023-01-31 23:02:22 +00:00
dependabot[bot] 7acd72b5ec
build(deps): bump toml from 0.5.11 to 0.6.0: changes `zebrad generate` to use double quoted strings, old configs still work (#6029)
* build(deps): bump toml from 0.5.11 to 0.6.0

Bumps [toml](https://github.com/toml-rs/toml) from 0.5.11 to 0.6.0.
- [Release notes](https://github.com/toml-rs/toml/releases)
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.5.11...toml-v0.6.0)

---
updated-dependencies:
- dependency-name: toml
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add production config in toml 0.6.0 format

* Add getblocktemplate feature config in toml 0.6.0 format

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: teor <teor@riseup.net>
2023-01-27 01:01:08 +00:00
teor b3a480757a
fix(rpc): Return detailed errors to the RPC client when a block proposal fails (#5993)
* Return detailed errors to the RPC client when a block proposal fails

* Change proposal reject type in acceptance test
2023-01-19 06:41:55 +00:00
teor 6a06cbf3ad
test(rpc): Create a script that submits block proposals to zcashd (#5944)
* Revert "Update code that we're going to delete in the next PR anyway"

This reverts commit 1fed70da9e.

* Initial zcash-test-block-template script without block proposal construction

* Try creating block data using jq and printf

* Move proposal_block_from_template() to zebra-rpc and remove eyre dependency

* Implement FromStr for DateTime32 and Duration32

* Basic block-template-to-proposal command without time source handling

* Move block proposal code into its own module

* Use time source in block-template-to-proposal

* Make block-template-to-proposal require the getblocktemplate-rpcs feature

* Use block-template-to-proposal in the test script zcash-rpc-block-template-to-proposal

* Apply new hex formatting to commitments and nonces in block proposal tests

* Re-add missing imports from rebase

* Re-add missing conversions from rebase

* Update to new method name after rebase

* Derive a Default impl

* Restore a comment that was accidentally deleted during the rebase

* Avoid a clippy::unwrap-in-result

* Temporarily fix the code for a disabled test

* Fix tool build with Docker caches

* Make clippy happy with the temporary fix

* Give a pass/fail status for each proposal response

* Accept zcashd block templates in block-template-to-proposal

* Fix pretty printing

* Zebra expects a longpollid, so give it a dummy value

* Add "required" fields which Zebra requires

* Use curtime as the dummy maxtime in zcashd templates

* Support large block proposals by reading proposal data from a file

* Test all valid time modes for each proposal

* Allow the user to set the time command

* Put debug logs into their own files

* Exit with an error status when any proposal is invalid

* Log the date and time to make it easier to match errors to node logs
2023-01-18 02:11:15 +00:00
teor 8c451968ee
fix(diagnostics): Hex-encode debug format of commitments, nonces, and nullifiers (#5960)
* Format commitments and nonces as hex

* Format Sprout and Sapling nullifiers as hex when debugging

* Format Sprout commitments as hex when debugging

* Format redpallas keys as hex when debugging

* Update code that we're going to delete in the next PR anyway
2023-01-17 13:57:22 +00:00
Arya 1bb8a9c924
change(rpc): Add getpeerinfo RPC method (#5951)
* adds ValidateBlock request to state

* adds `Request` enum in block verifier

skips solution check for BlockProposal requests

calls CheckBlockValidity instead of Commit block for BlockProposal requests

* uses new Request in references to chain verifier

* adds getblocktemplate proposal mode response type

* makes getblocktemplate-rpcs feature in zebra-consensus select getblocktemplate-rpcs in zebra-state

* Adds PR review revisions

* adds info log in CheckBlockProposalValidity

* Reverts replacement of match statement

* adds `GetBlockTemplate::capabilities` fn

* conditions calling checkpoint verifier on !request.is_proposal

* updates references to validate_and_commit_non_finalized

* adds snapshot test, updates test vectors

* adds `should_count_metrics` to NonFinalizedState

* Returns an error from chain verifier for block proposal requests below checkpoint height

adds feature flags

* adds "proposal" to GET_BLOCK_TEMPLATE_CAPABILITIES_FIELD

* adds back block::Request to zebra-consensus lib

* updates snapshots

* Removes unnecessary network arg

* skips req in tracing intstrument for read state

* Moves out block proposal validation to its own fn

* corrects `difficulty_threshold_is_valid` docs

adds/fixes some comments, adds TODOs

general cleanup from a self-review.

* Update zebra-state/src/service.rs

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Update zebra-rpc/src/methods/get_block_template_rpcs.rs

Co-authored-by: teor <teor@riseup.net>

* check best chain tip

* Update zebra-state/src/service.rs

Co-authored-by: teor <teor@riseup.net>

* Applies cleanup suggestions from code review

* updates gbt acceptance test to make a block proposal

* fixes json parsing mistake

* adds retries

* returns reject reason if there are no retries left

* moves result deserialization to RPCRequestClient method, adds docs, moves jsonrpc_core to dev-dependencies

* moves sleep(EXPECTED_TX_TIME) out of loop

* updates/adds info logs in retry loop

* Revert "moves sleep(EXPECTED_TX_TIME) out of loop"

This reverts commit f7f0926f4050519687a79afc16656c3f345c004b.

* adds `allow(dead_code)`

* tests with curtime, mintime, & maxtime

* Fixes doc comment

* Logs error responses from chain_verifier CheckProposal requests

* Removes retry loop, adds num_txs log

* removes verbose info log

* sorts mempool_txs before generating merkle root

* Make imports conditional on a feature

* Disable new CI tests until bugs are fixed

* adds support for getpeerinfo RPC

* adds vector test

* Always passes address_book to RpcServer

* Update zebra-network/src/address_book.rs

Co-authored-by: teor <teor@riseup.net>

* Renames PeerObserver to AddressBookPeers

* adds getpeerinfo acceptance test

* Asserts that addresses parse into SocketAddr

* adds launches_lightwalletd field to TestType::LaunchWithEmptyState and uses it from the get_peer_info test

* renames peer_observer mod

* uses SocketAddr as `addr` field type in PeerInfo

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2023-01-17 07:09:07 +00:00