2022-03-17 15:59:46 -07:00
|
|
|
//! Basic integration tests for zebra-state
|
|
|
|
|
|
|
|
use std::sync::Arc;
|
|
|
|
|
2020-06-22 21:31:44 -07:00
|
|
|
use color_eyre::eyre::Report;
|
|
|
|
use once_cell::sync::Lazy;
|
2022-03-17 15:59:46 -07:00
|
|
|
|
change(state): Write non-finalized blocks to the state in a separate thread, to avoid network and RPC hangs (#5257)
* Add a new block commit task and channels, that don't do anything yet
* Add last_block_hash_sent to the state service, to avoid database accesses
* Update last_block_hash_sent regardless of commit errors
* Rename a field to StateService.max_queued_finalized_height
* Commit finalized blocks to the state in a separate task
* Check for panics in the block write task
* Wait for the block commit task in tests, and check for errors
* Always run a proptest that sleeps once
* Add extra debugging to state shutdowns
* Work around a RocksDB shutdown bug
* Close the finalized block channel when we're finished with it
* Only reset state queue once per error
* Update some TODOs
* Add a module doc comment
* Drop channels and check for closed channels in the block commit task
* Close state channels and tasks on drop
* Remove some duplicate fields across StateService and ReadStateService
* Try tweaking the shutdown steps
* Update and clarify some comments
* Clarify another comment
* Don't try to cancel RocksDB background work on drop
* Fix up some comments
* Remove some duplicate code
* Remove redundant workarounds for shutdown issues
* Remode a redundant channel close in the block commit task
* Remove a mistaken `!force` shutdown condition
* Remove duplicate force-shutdown code and explain it better
* Improve RPC error logging
* Wait for chain tip updates in the RPC tests
* Wait 2 seconds for chain tip updates before skipping them
* Remove an unnecessary block_in_place()
* Fix some test error messages that were changed by earlier fixes
* Expand some comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
* Actually drop children of failed blocks
* Explain why we drop descendants of failed blocks
* Clarify a comment
* Wait for chain tip updates in a failing test on macOS
* Clean duplicate finalized blocks when the non-finalized state activates
* Send an error when receiving a duplicate finalized block
* Update checkpoint block behaviour, document its consensus rule
* Wait for chain tip changes in inbound_block_height_lookahead_limit test
* Wait for the genesis block to commit in the fake peer set mempool tests
* Disable unreliable mempool verification check in the send transaction test
* Appease rustfmt
* Use clear_finalized_block_queue() everywhere that blocks are dropped
* Document how Finalized and NonFinalized clones are different
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* updates comments, renames send_process_queued, other minor cleanup
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* updates comments, renames send_process_queued, other minor cleanup
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* removes duplicate field definitions on StateService that were a result of a bad merge
* update NotReadyToBeCommitted error message
* Appear rustfmt
* Fix doc links
* Rename a function to initial_contextual_validity()
* Do error tasks on Err, and success tasks on Ok
* Simplify parent_error_map truncation
* Rewrite best_tip() to use tip()
* Rename latest_mem() to latest_non_finalized_state()
```sh
fastmod latest_mem latest_non_finalized_state zebra*
cargo fmt --all
```
* Simplify latest_non_finalized_state() using a new WatchReceiver API
* Expand some error messages
* Send the result after updating the channels, and document why
* wait for chain_tip_update before cancelling download in mempool_cancel_mined
* adds `sent_non_finalized_block_hashes` field to StateService
* adds batched sent_hash insertions and checks sent hashes in queue_and_commit_non_finalized before adding a block to the queue
* check that the `curr_buf` in SentHashes is not empty before pushing it to the `sent_bufs`
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix rustfmt
* Check for finalized block heights using zs_contains()
* adds known_utxos field to SentHashes
* updates comment on SentHashes.add method
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* return early when there's a duplicate hash in QueuedBlocks.queue instead of panicking
* Make finalized UTXOs near the final checkpoint available for full block verification
* Replace a checkpoint height literal with the actual config
* Update mainnet and testnet checkpoints - 7 October 2022
* Fix some state service init arguments
* Allow more lookahead in the downloader, but less lookahead in the syncer
* Add the latest config to the tests, and fix the latest config check
* Increase the number of finalized blocks checked for non-finalized block UTXO spends
* fix(log): reduce verbose logs for block commits (#5348)
* Remove some verbose block write channel logs
* Only warn about tracing endpoint if the address is actually set
* Use CloneError instead of formatting a non-cloneable error
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Increase block verify timeout
* Work around a known block timeout bug by using a shorter timeout
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-10-11 12:25:45 -07:00
|
|
|
use zebra_chain::{
|
|
|
|
block::{Block, Height},
|
|
|
|
parameters::Network,
|
|
|
|
serialization::ZcashDeserialize,
|
|
|
|
};
|
2021-06-03 15:48:40 -07:00
|
|
|
use zebra_test::transcript::{ExpectedTranscriptError, Transcript};
|
2020-06-22 21:31:44 -07:00
|
|
|
|
|
|
|
use zebra_state::*;
|
|
|
|
|
2021-06-03 15:48:40 -07:00
|
|
|
static COMMIT_FINALIZED_BLOCK_MAINNET: Lazy<
|
|
|
|
Vec<(Request, Result<Response, ExpectedTranscriptError>)>,
|
|
|
|
> = Lazy::new(|| {
|
|
|
|
let block: Arc<_> =
|
|
|
|
Block::zcash_deserialize(&zebra_test::vectors::BLOCK_MAINNET_GENESIS_BYTES[..])
|
|
|
|
.unwrap()
|
|
|
|
.into();
|
|
|
|
let block2 = block.clone();
|
|
|
|
let hash = block.hash();
|
|
|
|
vec![
|
|
|
|
(
|
2023-06-01 05:29:03 -07:00
|
|
|
Request::CommitCheckpointVerifiedBlock(block.into()),
|
2021-06-03 15:48:40 -07:00
|
|
|
Ok(Response::Committed(hash)),
|
|
|
|
),
|
|
|
|
(
|
|
|
|
Request::Block(hash.into()),
|
|
|
|
Ok(Response::Block(Some(block2))),
|
|
|
|
),
|
|
|
|
]
|
|
|
|
});
|
2020-09-04 02:50:25 -07:00
|
|
|
|
2021-06-03 15:48:40 -07:00
|
|
|
static COMMIT_FINALIZED_BLOCK_TESTNET: Lazy<
|
|
|
|
Vec<(Request, Result<Response, ExpectedTranscriptError>)>,
|
|
|
|
> = Lazy::new(|| {
|
|
|
|
let block: Arc<_> =
|
|
|
|
Block::zcash_deserialize(&zebra_test::vectors::BLOCK_TESTNET_GENESIS_BYTES[..])
|
|
|
|
.unwrap()
|
|
|
|
.into();
|
|
|
|
let block2 = block.clone();
|
|
|
|
let hash = block.hash();
|
|
|
|
vec![
|
|
|
|
(
|
2023-06-01 05:29:03 -07:00
|
|
|
Request::CommitCheckpointVerifiedBlock(block.into()),
|
2021-06-03 15:48:40 -07:00
|
|
|
Ok(Response::Committed(hash)),
|
|
|
|
),
|
|
|
|
(
|
|
|
|
Request::Block(hash.into()),
|
|
|
|
Ok(Response::Block(Some(block2))),
|
|
|
|
),
|
|
|
|
]
|
|
|
|
});
|
2020-06-22 21:31:44 -07:00
|
|
|
|
2022-07-21 16:16:41 -07:00
|
|
|
#[tokio::test(flavor = "multi_thread")]
|
2020-07-26 16:57:18 -07:00
|
|
|
async fn check_transcripts_mainnet() -> Result<(), Report> {
|
2020-08-15 15:45:37 -07:00
|
|
|
check_transcripts(Network::Mainnet).await
|
2020-07-26 16:57:18 -07:00
|
|
|
}
|
|
|
|
|
2022-07-21 16:16:41 -07:00
|
|
|
#[tokio::test(flavor = "multi_thread")]
|
2020-07-26 16:57:18 -07:00
|
|
|
async fn check_transcripts_testnet() -> Result<(), Report> {
|
2024-04-16 19:20:34 -07:00
|
|
|
check_transcripts(Network::new_default_testnet()).await
|
2020-07-23 04:12:20 -07:00
|
|
|
}
|
|
|
|
|
2020-07-17 15:19:00 -07:00
|
|
|
#[spandoc::spandoc]
|
2020-07-26 16:57:18 -07:00
|
|
|
async fn check_transcripts(network: Network) -> Result<(), Report> {
|
2022-08-04 08:44:44 -07:00
|
|
|
let _init_guard = zebra_test::init();
|
2020-06-22 21:31:44 -07:00
|
|
|
|
2020-09-09 17:59:58 -07:00
|
|
|
let mainnet_transcript = &[&COMMIT_FINALIZED_BLOCK_MAINNET];
|
|
|
|
let testnet_transcript = &[&COMMIT_FINALIZED_BLOCK_TESTNET];
|
2020-08-04 13:39:32 -07:00
|
|
|
|
2024-03-05 06:12:25 -08:00
|
|
|
let net_data = if network.is_mainnet() {
|
|
|
|
mainnet_transcript
|
|
|
|
} else {
|
|
|
|
testnet_transcript
|
|
|
|
};
|
|
|
|
|
|
|
|
for transcript_data in net_data {
|
change(state): Write non-finalized blocks to the state in a separate thread, to avoid network and RPC hangs (#5257)
* Add a new block commit task and channels, that don't do anything yet
* Add last_block_hash_sent to the state service, to avoid database accesses
* Update last_block_hash_sent regardless of commit errors
* Rename a field to StateService.max_queued_finalized_height
* Commit finalized blocks to the state in a separate task
* Check for panics in the block write task
* Wait for the block commit task in tests, and check for errors
* Always run a proptest that sleeps once
* Add extra debugging to state shutdowns
* Work around a RocksDB shutdown bug
* Close the finalized block channel when we're finished with it
* Only reset state queue once per error
* Update some TODOs
* Add a module doc comment
* Drop channels and check for closed channels in the block commit task
* Close state channels and tasks on drop
* Remove some duplicate fields across StateService and ReadStateService
* Try tweaking the shutdown steps
* Update and clarify some comments
* Clarify another comment
* Don't try to cancel RocksDB background work on drop
* Fix up some comments
* Remove some duplicate code
* Remove redundant workarounds for shutdown issues
* Remode a redundant channel close in the block commit task
* Remove a mistaken `!force` shutdown condition
* Remove duplicate force-shutdown code and explain it better
* Improve RPC error logging
* Wait for chain tip updates in the RPC tests
* Wait 2 seconds for chain tip updates before skipping them
* Remove an unnecessary block_in_place()
* Fix some test error messages that were changed by earlier fixes
* Expand some comments, fix typos
Co-authored-by: Marek <mail@marek.onl>
* Actually drop children of failed blocks
* Explain why we drop descendants of failed blocks
* Clarify a comment
* Wait for chain tip updates in a failing test on macOS
* Clean duplicate finalized blocks when the non-finalized state activates
* Send an error when receiving a duplicate finalized block
* Update checkpoint block behaviour, document its consensus rule
* Wait for chain tip changes in inbound_block_height_lookahead_limit test
* Wait for the genesis block to commit in the fake peer set mempool tests
* Disable unreliable mempool verification check in the send transaction test
* Appease rustfmt
* Use clear_finalized_block_queue() everywhere that blocks are dropped
* Document how Finalized and NonFinalized clones are different
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* updates comments, renames send_process_queued, other minor cleanup
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* sends non-finalized blocks to the block write task
* passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_in_finalized_chain instead of FinalizedState
* updates comments, renames send_process_queued, other minor cleanup
* Update zebra-state/src/service/write.rs
Co-authored-by: teor <teor@riseup.net>
* update assert_block_can_be_validated comment
* removes `mem` field from StateService
* removes `disk` field from StateService and updates block_iter to use `ZebraDb` instead of the finalized state
* updates tests that use the disk to use read_service.db instead
* moves best_tip to a read fn and returns finalized & non-finalized states from setup instead of the state service
* changes `contextual_validity` to get the network from the finalized_state instead of another param
* swaps out StateService with FinalizedState and NonFinalizedState in tests
* adds NotReadyToBeCommitted error and returns it from validate_and_commit when a blocks parent hash is not in any chain
* removes NonFinalizedWriteCmd and calls, moves update_latest_channels above rsp_tx.send
* makes parent_errors_map an indexmap
* clears non-finalized block queue when the receiver is dropped and when the StateService is being dropped
* removes duplicate field definitions on StateService that were a result of a bad merge
* update NotReadyToBeCommitted error message
* Appear rustfmt
* Fix doc links
* Rename a function to initial_contextual_validity()
* Do error tasks on Err, and success tasks on Ok
* Simplify parent_error_map truncation
* Rewrite best_tip() to use tip()
* Rename latest_mem() to latest_non_finalized_state()
```sh
fastmod latest_mem latest_non_finalized_state zebra*
cargo fmt --all
```
* Simplify latest_non_finalized_state() using a new WatchReceiver API
* Expand some error messages
* Send the result after updating the channels, and document why
* wait for chain_tip_update before cancelling download in mempool_cancel_mined
* adds `sent_non_finalized_block_hashes` field to StateService
* adds batched sent_hash insertions and checks sent hashes in queue_and_commit_non_finalized before adding a block to the queue
* check that the `curr_buf` in SentHashes is not empty before pushing it to the `sent_bufs`
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* Fix rustfmt
* Check for finalized block heights using zs_contains()
* adds known_utxos field to SentHashes
* updates comment on SentHashes.add method
* Apply suggestions from code review
Co-authored-by: teor <teor@riseup.net>
* return early when there's a duplicate hash in QueuedBlocks.queue instead of panicking
* Make finalized UTXOs near the final checkpoint available for full block verification
* Replace a checkpoint height literal with the actual config
* Update mainnet and testnet checkpoints - 7 October 2022
* Fix some state service init arguments
* Allow more lookahead in the downloader, but less lookahead in the syncer
* Add the latest config to the tests, and fix the latest config check
* Increase the number of finalized blocks checked for non-finalized block UTXO spends
* fix(log): reduce verbose logs for block commits (#5348)
* Remove some verbose block write channel logs
* Only warn about tracing endpoint if the address is actually set
* Use CloneError instead of formatting a non-cloneable error
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Increase block verify timeout
* Work around a known block timeout bug by using a shorter timeout
Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-10-11 12:25:45 -07:00
|
|
|
// We're not verifying UTXOs here.
|
2024-03-19 13:45:27 -07:00
|
|
|
let (service, _, _, _) = zebra_state::init(Config::ephemeral(), &network, Height::MAX, 0);
|
2020-06-22 21:31:44 -07:00
|
|
|
let transcript = Transcript::from(transcript_data.iter().cloned());
|
2020-07-17 15:19:00 -07:00
|
|
|
/// SPANDOC: check the on disk service against the transcript
|
2020-06-22 21:31:44 -07:00
|
|
|
transcript.check(service).await?;
|
|
|
|
}
|
|
|
|
|
|
|
|
Ok(())
|
|
|
|
}
|