Commit Graph

107 Commits

Author SHA1 Message Date
Alfredo Garcia 8075d616fb
feat(mempool): add ZIP-317 rules to mempool (#6556)
* add ZIP-317 rules to mempool

* fix some docs

* rustfmt

* fix import

* typo

* fix tests

* fix tests 2

* fix tests 3

* fix tests 4

* fix tests 5

* move constant

* fix constant for feature

* document/quote zip rules

* add Minimum Fee Rate rule

* change(mempool): Refactor the ZIP-317 minimum fee rate calculation to use usize (#6585)

* Refactor the minimum fee rate calculation to use usize

* Check for overflow if constants change

* remove 1 rule check, fix docs

---------

Co-authored-by: teor <teor@riseup.net>
2023-05-02 00:13:33 +00:00
teor 166526a088
feat(ui): Add a terminal-based progress bar to Zebra (#6235)
* Implement Display and to_string() for NetworkUpgrade

* Add a progress-bar feature to zebrad

* Add the progress bar writer to the tracing component

* Add a block progress bar transmitter

* Correctly shut down the progress bar, and shut it down on an interrupt

* Make it clearer that the progress task never exits

* Add a config for writing logs to a file

* Add a progress-bar feature to zebra-network

* Add a progress bar for the address book size

* Add progress bars for never attempted and failed peers

* Add an optional limit and label to connection counters

* Add open connection progress bars

* Improve CheckpointList API and CheckpointVerifier debugging

* Add checkpoint index and checkpoint queue progress bars

* Security: Limit the number of non-finalized chains tracked by Zebra

* Make some NonFinalizedState methods available with proptest-impl

* Add a non-finalized chain count progress bar

* Track the last fork height for newly forked chains

* Add a should_count_metrics to Chain

* Add a display method for PartialCumulativeWork

* Add a progress bar for each chain fork

* Add a NonFinalizedState::disable_metrics() method and switch to using it

* Move metrics out of Chain because we can't update Arc<Chain>

* Fix: consistently use best chain order when searching chains

* Track Chain progress bars in NonFinalizedState

* Display work as bits, not a multiple of the target difficulty

* Handle negative fork lengths by reporting "No fork"

* Correctly disable unused fork bars

* clippy: rewrite using `match _.cmp(_) { ... }`

* Initial mempool progress bar implementation

* Update Cargo.lock

* Add the actual transaction size as a description to the cost bar

* Only show mempool progress bars after first activation

* Add queued and rejected mempool progress bars

* Clarify cost note is actual size

* Add tracing.log_file config and progress-bar feature to zebrad docs

* Derive Clone for Chain

* Upgrade to howudoin 0.1.2 and remove some bug workarounds

* Directly call the debug formatter to Display a Network

Co-authored-by: Arya <aryasolhi@gmail.com>

* Rename the address count metric to num_addresses

Co-authored-by: Arya <aryasolhi@gmail.com>

* Simplify reverse checkpoint lookup

Co-authored-by: Arya <aryasolhi@gmail.com>

* Simplify progress bar shutdown code

Co-authored-by: Arya <aryasolhi@gmail.com>

* Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE

* Document that the progress task runs forever

* Fix progress log formatting

* If progress-bar is on, log to a file by default

* Create missing directories for log files

* Add file security docs for running Zebra with elevated permissions

* Document automatic log file, spell progress-bar correctly

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-04-13 08:42:17 +00:00
teor 8390e4e0cd
refactor(state): Simplify difficulty and median-time-past state and mempool requests (#6031)
* Clarify function docs, rename variables, and fix log typos

* Add a ReadState best chain clone method, but don't use it yet

* Use the new calculate_median_time_past() function in existing code

* Skip a state request if the lock time is a height

* Remove dummy arguments and extra blocks from median-time-past calculation

* Update tests to remove requests that are no longer sent

* Simplify getting the best chain

Co-authored-by: Arya <aryasolhi@gmail.com>

* Clarify some function docs

Co-authored-by: Arya <aryasolhi@gmail.com>

* assigns `next_median_time_past` value from if statement

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-01-31 20:42:11 +00:00
teor cb1045ae5f
change(mempool): Re-verify mempool transactions after a chain fork, rather than re-downloading them all (#5841)
* Move Drop from mempool::ActiveState to mempool::Downloads, to avoid bugs

* Re-verify mempool transactions after a fork

And add a marker struct for mempool download cancellation.

* Update README based on recent mitigations for some issues, tidy format

* Make mempool proptests easier to debug

* Make UnminedTx Display text much smaller

* Update tests for mempool transaction re-verification after forks

* Retry all stored and pending transactions

* Fix a test to check for mempool reset retries
2022-12-12 23:19:45 +00:00
teor 4078e244d3
fix(lint): Box large error types to resolve the clippy large result err variant lint (#5759)
* Box errors to deal with large error warnings, add accessor methods for error properties

* Remove or explain some large enum variant lints

* Turn some tickets into TODOs

* Allow missing docs on single-field error enum variants

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-12-08 06:11:33 +00:00
teor 7e13677197
change(rpc): generate coinbase transactions in the getblocktemplate RPC (#5580)
* Add a getblocktemplate-rpcs feature to zebra-chain, and fix missing feature deps

* Add a coinbase transaction creation stub

* Add coinbase creation to zebra-chain

* Add coinbase creation and miner subsidy to zebra-consensus

* Add the miner config to the GetBlockTemplateRpcImpl

* Generate the coinbase transaction in the getblocktemplate RPC

* Provide fake valid block heights to getblocktemplate RPC tests

* Update getblocktemplate RPC snapshots

* Add a getblocktemplate.coinbase_tx deserialized transaction snapshot test

* Update snapshots

* Return funding stream outputs in the same order every time

* Update snapshots

* Fix a script bytes bug

* Update snapshots
2022-11-10 00:12:27 +00:00
Alfredo Garcia 233220ace0
remove dead code from zebra-chain (#5464)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-10-25 03:23:50 +00:00
Alfredo Garcia 97fb85dca9
lint(clippy): add `unwrap_in_result` lint (#4667)
* `unwrap_in_result` in zebra-chain crate

* `unwrap_in_result` in zebra-script crate

* `unwrap_in_result` in zebra-state crate

* `unwrap_in_result` in zebra-consensus crate

* `unwrap_in_result` in zebra-test crate

* `unwrap_in_result` in zebra-network crate

* `unwrap_in_result` in zebra-rpc crate

* `unwrap_in_result` in zebrad crate

* rustfmt

* revert `?` and add exceptions

* explain some panics better

* move some lint positions

* replace a panic with error

* Fix rustfmt?

Co-authored-by: teor <teor@riseup.net>
2022-06-28 06:22:07 +00:00
Marek b8b35f8da9
fix(doc): Fix various doc warnings, part 1 (#4514)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fa.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-06-02 15:07:35 +00:00
Marek 6f896ef5a5
fix(doc): Fix the syntax of links in comments (#4494)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fa.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-05-30 20:12:11 +00:00
Conrado Gouvea dff25473aa
refactor: document coinbase rules, refactor to ease understanding (#4056)
* refactor: document coinbase rules, refactor to ease understanding

* Update zebra-consensus/src/block/check.rs

Co-authored-by: teor <teor@riseup.net>

* remove no longer used contains_prevout_input()

* remove unused CoinbaseHasPrevOutInput

* update coinbase description in documentation

Co-authored-by: teor <teor@riseup.net>
2022-04-20 09:31:12 +00:00
Conrado Gouvea e7c0a78d4d
feat(rpc): add getrawtransaction (#3908)
* feat(rpc): add getrawtransaction

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Apply suggestions from code review

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* address review comments

* move SerializedTransaction to the right module

* Use try_into() instead of 'as'

* add proptests

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-24 09:45:37 +00:00
teor b4deca2912
6. add(test): add more tests for height and transaction location serialization (#3879)
* Add round-trip test for OutputLocation disk serialization

* Make the transaction snapshot tests more accurate

Previously, we were showing the genesis transaction hash at height 0, index 0.

But its hash is actually not stored by location in the database,
because the genesis transaction is skipped due to a consensus rule.

* Update the transaction snapshot data

* Add history tree snapshot tests

At the current test heights, the history trees are all empty.

* Add the history tree snapshot data

* Update comments

* Simplify snapshot test code

* Make some serde::Serialize impls test-only, so we can hex-encode them

This should also speed up release compilation a bit.

* Add snapshot test code for UTXOs

* Add snapshot data for UTXOs
2022-03-18 20:30:16 +00:00
Alfredo Garcia 29ad801a35
docs: Transaction consensus rules: Size rules (#3461)
* refactor transaction size consensus rules

* quote mssing consensus rule

* nit

Co-authored-by: teor <teor@riseup.net>

* move consensus rule doc

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-08 20:28:40 +00:00
Conrado Gouvea f270fd2de6
Prepare for changes in ZIP-244 (#3415)
* Add all_previous_outputs; load UTXOs in transaction verifier

* Remove UTXO loading and returning from script.rs

* Don't pass state service to script verifier

* Remove output from is_valid()

* Refactor loading UTXOs to separate function

* Pass all_previous_output to sighash

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Create AwaitUtxo only when needed; formatting

* Add comments about output vectors in tests

* Change sighash() to receive reference and avoid cloning

* Expand comments

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-01-31 15:28:42 +00:00
Conrado Gouvea 6ec42c6044
Integrate JoinSplit verifier (#3180)
* Integrate JoinSplit verifier with transaction verifier

* Add test with malformed Groth16 Output proof

* Use TryFrom instead of From in ItemWrapper to correctly propagate malformed proof errors

* Simplify by removing ItemWrapper and directly TryFrom into Item

* Fix existing tests to work with JoinSplit validation

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com>
2021-12-13 14:50:49 -05:00
Conrado Gouvea ee073c0876
Validate JoinSplit proofs (#3128)
* Validate JoinSplit proofs

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Move primary input encoding to zebra_consensus

* Improve typing of h_sig; add RandomSeed

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-12-10 16:33:15 +00:00
Deirdre Connolly b973b7a622
Checking of Sprout anchors in non-finalized state (#3123)
* Do prelim checking of Sprout anchors in non-finalized state

Does not check intra-transaction interstitial states yet

* Populate sprout anchors to allow other state tests to pass

* Preliminary interstitial sprout note commitment tree anchor checks implementation

* Make sure only prior anchors are checked in the same transaction

* Add tests

* Refactor a comment

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Use the first `JoinSplit`s from mainnet

* Print debug messages

* Use correct blocks for the tests

Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-12-09 16:50:26 +00:00
teor 4ce6fbccc4
Fix new clippy lints in clippy nightly (#3176) 2021-12-09 14:19:14 +00:00
Deirdre Connolly e6ffe374d4
Validate sapling, orchard anchors (#3084)
* Add Transaction::sprout_joinsplits()

* Add Anchor variants to ValidateContextError

* Make Chain anchor collections pub(crate)

* tracing::instrument several methods in state

* Add contains_*_anchors methods to FinalizedState

* Add check::anchors module and function

* Verify that anchors_refer_to_earlier_treestates in when updating chains in non-finalized state

* Update zebra-state/src/service/check/anchors.rs

Co-authored-by: teor <teor@riseup.net>

* Add anchors() to sapling::ShieldedData

* Add sapling_anchors() to Transaction

* Use Transaction::sapling_anchors() in the anchors_refer_to_earlier_treestates() check

* Whoops, itertools

* Add a comment for improvement

Co-authored-by: teor <teor@riseup.net>

* Add & use a cfg(test) method on FinalizedState to prep test state with anchors to allow other tests to pass contextual checks

* Allow test nullifier checks to pass by populating anchor sets, allowing test anchor checks to pass

* Add mainnet block 419202 and its sapling note commitment tree root to test vectors

* Test sapling anchor verification using the first few Sapling blocks data

* Correct comment

* assert_eq instead of assert(bool)

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Update zebra-state/src/service/non_finalized_state.rs

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Marek <mail@marek.onl>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-11-30 16:05:35 +00:00
Marek 3c9ad89018
Add Sprout anchors to `zebra-state` (#3100)
* Add Sprout anchors to the state

* Update zebra-state/src/service/non_finalized_state/chain.rs

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Return new types of note commitments from Sprout transactions

* Refactor the tests

* Refactor some comments

Co-authored-by: teor <teor@riseup.net>

* Increment `DATABASE_FORMAT_VERSION`

* Update `test.yml` with the new image name

* Refactor the `version = 5` transaction description

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Update comment

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-11-30 11:05:58 +01:00
Alfredo Garcia a61eae0065
Validate miner transaction fees (#3067)
* validate consensus rule: negative fee not allowed

* fix a test TODO

* fix imports

* move import back

* fix panic text

* join consensus rule check code

* match assertion better in tests

* fix test

* fix consensus rule validation

* remove panics

* Delete a TODO

Co-authored-by: teor <teor@riseup.net>
2021-11-24 03:36:17 +00:00
Janito Vaqueiro Ferreira Filho ec2c980bb1
Validate transaction lock times (#3060)
* Create a `LockTime::unlocked` helper constructor

Returns a `LockTime` that is unlocked at the genesis block.

* Return `Option<LockTime>` from `lock_time` method

Prepare to return `None` for when a transaction has its lock time
disabled.

* Return `None` instead of zero `LockTime`

Because a zero lock time means that the transaction was unlocked at the
genesis block, so it was never actually locked.

* Rephrase zero lock time check comment

Clarify that the check is not redundant, and is necessary for the
genesis transaction.

Co-authored-by: teor <teor@riseup.net>

* Add a `transparent::Input::sequence` getter method

Retrieve a transparent input's sequence number.

* Check if lock time is enabled by a sequence number

Validate the consensus rule that the lock time is only enabled if at
least one transparent input has a value different from `u32::MAX` as its
sequence number.

* Add more Zcash specific details to comment

Explain the Zcash specific lock time behaviors.

Co-authored-by: teor <teor@riseup.net>

* Add `time` field to `Request::Block` variant

The block time to use to check if the transaction was unlocked and
allowed to be included in the block.

* Add `Request::block_time` getter

Returns the block time for the block that owns the transaction being
validated or the current time plus a tolerance for mempool transactions.

* Validate transaction lock times

If they are enabled by a transaction's transparent input sequence
numbers, make sure that they are in the past.

* Add comments with consensus rule parts

Make it easier to map what part of the consensus rule each match arm is
responsible for.

Co-authored-by: teor <teor@riseup.net>
2021-11-23 05:53:53 +00:00
Marek 8963007397
Sprout note commitment trees (#3051)
* Implement incremental note commitment Merkle tree for Sprout

* Add tests for Sprout note commitment tree

* Remove the `Arbitrary` attribute

* Reverse the vector of empty roots

* Add more tests

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* rustdoc

* Rustdoc

* rustdoc links

* Oops, need the trait in scope to use it

* Avoid accessing the wrapped hash directly

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* rustfmt

* Add typing

* Avoid accessing the wrapped hash directly

* Implement incremental note commitment Merkle tree for Sprout

* Add tests for Sprout note commitment tree

* Remove the `Arbitrary` attribute

* Reverse the vector of empty roots

* Add more tests

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Refactor rustdoc

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* rustdoc

* Rustdoc

* rustdoc links

* Oops, need the trait in scope to use it

* Avoid accessing the wrapped hash directly

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* rustfmt

* Add typing

* Avoid accessing the wrapped hash directly

* Add Overwinter final roots (test vectors)

* Test sprout note commitments trees on Overwinter blocks

* Add new test vectors

* Finish the tests for the note commitment trees

* Make the wrapped hash in `Root` private

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Deirdre Connolly <durumcrustulum@gmail.com>
2021-11-18 23:05:52 +00:00
Alfredo Garcia f7c1907fb6
validate orchard flags in v5 (#3035) 2021-11-08 21:45:54 +00:00
teor 2d129414e0
Store the transaction fee in the mempool storage (#2885)
* Create a new VerifiedUnminedTx containing the miner fee

* Use VerifiedUnminedTx in mempool verification responses

And do a bunch of other cleanups.

* Use VerifiedUnminedTx in mempool download and verifier

* Use VerifiedUnminedTx in mempool storage and verified set

* Impl Display for VerifiedUnminedTx, and some convenience methods

* Use VerifiedUnminedTx in existing tests
2021-10-18 11:24:37 +10:00
Janito Vaqueiro Ferreira Filho 9e78a8af40
Refactor mempool spend conflict checks to increase performance (#2826)
* Add `HashSet`s to help spend conflict detection

Keep track of the spent transparent outpoints and the revealed
nullifiers.

Clippy complained that the `ActiveState` had variants with large size
differences, but that was expected, so I disabled that lint on that
`enum`.

* Clear the `HashSet`s when clearing the mempool

Clear them so that they remain consistent with the set of verified
transactions.

* Use `HashSet`s to check for spend conflicts

Store new outputs into its respective `HashSet`, and abort if a
duplicate output is found.

* Remove inserted outputs when aborting

Restore the `HashSet` to its previous state.

* Remove tracked outputs when removing a transaction

Keep the mempool storage in a consistent state when a transaction is
removed.

* Remove tracked outputs when evicting from mempool

Ensure eviction also keeps the tracked outputs consistent with the
verified transactions.

* Refactor to create a `VerifiedSet` helper type

Move the code to handle the output caches into the new type. Also move
the eviction code to make things a little simpler.

* Refactor to have a single `remove` method

Centralize the code that handles the removal of a transaction to avoid
mistakes.

* Move mempool size limiting back to `Storage`

Because the evicted transactions must be added to the rejected list.

* Remove leftover `dbg!` statement

Leftover from some temporary testing code.

Co-authored-by: teor <teor@riseup.net>

* Remove unnecessary `TODO`

It is more speculation than planning, so it doesn't add much value.

Co-authored-by: teor <teor@riseup.net>

* Fix typo in documentation

The verb should match the subject "transactions" which is plural.

Co-authored-by: teor <teor@riseup.net>

* Add a comment to warn about correctness

There's a subtle but important detail in the implementation that should
be made more visible to avoid mistakes in the future.

Co-authored-by: teor <teor@riseup.net>

* Remove outdated comment

Left-over from the attempt to move the eviction into the `VerifiedSet`.

* Improve comment explaining lint removal

Rewrite the comment explaining why the Clippy lint was ignored.

* Check for spend conflicts in `VerifiedSet`

Refactor to avoid API misuse.

* Test rejected transaction rollback

Using two transactions, perform the same test adding a conflict to both
of them to check if the second inserted transaction is properly
rejected. Then remove any conflicts from the second transaction and add
it again. That should work, because if it doesn't it means that when the
second transaction was rejected it left things it shouldn't in the
cache.

* Test removal of multiple transactions

When removing multiple transactions from the mempool storage, all of the
ones requested should be removed and any other transaction should be
still be there afterwards.

* Increase mempool size to 4, so that spend conflict tests work

If the mempool size is smaller than 4,
these tests don't fail on a trivial removal bug.
Because we need a minimum number of transactions in the mempool
to trigger the bug.

Also commit a proptest seed that fails on a trivial removal bug.
(This seed fails if we remove indexes in order,
because every index past the first removes the wrong transaction.)

* Summarise transaction data in proptest error output

* Summarise spend conflict field data in proptest error output

* Summarise multiple removal field data in proptest error output

And replace the very large proptest debug output with the new summary.

Co-authored-by: teor <teor@riseup.net>
2021-10-10 23:54:46 +00:00
Alfredo Garcia 37595c4b32
Mempool support for transaction expiration (#2774)
* mempool - support transaction expiration

* use `LatestChainTip` instead of state call

* clippy

* remove spawn task

* remove non needed async from function

* remove return value

* add a `expiry_height_mut()` method to `Transaction` for testing purposes

* fix `remove_expired_transactions()`

* add a `mempool_transaction_expiration()` test

* tidy cleanup to `expiry_height()`

* improve docs

* fix the build

* try fix macos build

* extend tests

* add doc to function

* clippy

* fix build

* start tests at block two
2021-09-29 16:52:44 +00:00
Janito Vaqueiro Ferreira Filho a0d45c38f3
Reject conflicting mempool transactions (#2765)
* Add `Transaction::spent_outpoints` getter method

Returns an iterator over the UTXO `OutPoint`s spent by the transaction.

* Add `mempool::Error::Conflict` variant

An error representing that a transaction was rejected because it
conflicts with another transaction that's already in the mempool.

* Reject conflicting mempool transactions

Reject including a transaction in the mempool if it spends outputs
already spent by, or reveals nullifiers already revealed by another
transaction in the mempool.

* Fix typo in documentation

Remove the `r` that was incorrectly added.

Co-authored-by: teor <teor@riseup.net>

* Specify that the conflict is a spend conflict

Make the situation clearer, because there are other types of conflict.

Co-authored-by: teor <teor@riseup.net>

* Clarify that the outpoints are from inputs

Because otherwise it could lead to confusion because it could also mean
the outputs of the transaction represented as `OutPoint` references.

Co-authored-by: teor <teor@riseup.net>

* Create `storage::tests::vectors` module

Refactor to follow the convention used for other tests.

* Add an `AtLeastOne::first_mut` method

A getter to allow changing the first element.

* Add an `AtLeastOne::push` method

Allow appending elements to the collection.

* Derive `Arbitrary` for `FieldNotPresent`

This is just to make the code that generates arbitrary anchors a bit
simpler.

* Test if conflicting transactions are rejected

Generate two transactions (either V4 or V5) and insert a conflicting
spend, which can be either a transparent UTXO, or a nullifier for one of
the shielded pools. Check that any attempt to insert both transactions
causes one to be accepted and the other to be rejected.

* Delete a TODO comment that we decided not to do

Co-authored-by: teor <teor@riseup.net>
2021-09-28 01:03:08 +00:00
Alfredo Garcia 825da448c0
Handle zero transaction expirations (#2782)
* handle zero transaction expirations

* add consensus rule doc

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
2021-09-22 13:44:52 +10:00
Conrado Gouvea 1ccb2de7c7
Add transaction downloader and verifier (#2679)
* Add transaction downloader

* Changed mempool downloader to be like inbound

* Verifier working (logs result)

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Fix coinbase check for mempool, improve is_coinbase() docs

* Change other downloads.rs docs to reflect the mempool downloads.rs changes

* Change TIMEOUTs to downloads.rs; add docs

* Renamed is_coinbase() to has_valid_coinbase_transaction_inputs() and contains_coinbase_input() to has_any_coinbase_inputs(); reorder checks

* Validate network upgrade for V4 transactions; check before computing sighash (for V5 too)

* Add block_ prefix to downloads and verifier

* Update zebra-consensus/src/transaction.rs

Co-authored-by: teor <teor@riseup.net>

* Add consensus doc; add more Block prefixes

Co-authored-by: teor <teor@riseup.net>
2021-09-02 00:06:20 +00:00
teor 2ed6679069
Use unmined types for transaction verifier mempool requests and responses (#2666)
* Add `Arc<Transaction>` conversions for Transaction IDs

* Use UnminedTxId as the transaction verifier response type

* Use UnminedTx in transaction verifier mempool requests

* Refactor is_mempool into a transaction verifier request method

* Order derives in alphabetical order

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-25 21:02:47 +00:00
teor 84c5f6189d
Create types for unmined transactions and their IDs (#2634)
* Create new types for unmined transactions and their IDs

* Add accessor methods for the parts of an unmined transaction ID
2021-08-18 15:52:42 +10:00
teor 6c86c8dd0d
Implement a WtxId struct, and use it in Zebra's external network protocol (#2618)
* Make the `AuthDigest` display order match transaction IDs

And derive `Hash`, just like transaction IDs.

Don't derive `serde` for now, because it's not needed.

* Move transaction::Hash test to tests module

* Add a simple AuthDigest display order test

* Add a WtxId type for wide transaction IDs

* Add conversions between transaction IDs and bytes

* Use the WtxId type in external network protocol messages
2021-08-16 21:26:08 +00:00
Conrado Gouvea eadca72e75
Implement ZIP-244 authorizing data commitment (auth_digest) (#2547)
* Implement ZIP-244 authorizing data commitment (auth_digest)

* s/Merke/Merkle/

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Fix AuthDataRoot computation to use padded leaves; add tests

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-13 12:58:04 -04:00
teor 298ececabe
Refactor value balances to support generated data modification (#2596)
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-10 14:42:02 +00:00
teor f09f2a9022
Check remaining transaction value & make value balance signs match the spec (#2566)
* Make Amount arithmetic more generic

To modify generated amounts, we need some extra operations on `Amount`.

We also need to extend existing operations to both `NonNegative` and
`NegativeAllowed` amounts.

* Add a constrain method for ValueBalance

* Derive Eq for ValueBalance

* impl Neg for ValueBalance

* Make some Amount arithmetic expectations explicit

* Explain why we use i128 for multiplication

And expand the overflow error details.

* Expand Amount::sum error details

* Make amount::Error field order consistent

* Rename an amount::Error variant to Constraint, so it's clearer

* Add specific pool variants to ValueBalanceError

* Update coinbase remaining value consensus rule comment

This consensus rule was updated recently to include coinbase transactions,
but Zebra doesn't check block subsidy or miner fees yet.

* Add test methods for modifying transparent values and shielded value balances

* Temporarily set values and value balances to zero in proptests

In both generated chains and proptests that construct their own transactions.

Using zero values reduces value calculation and value check test coverage.
A future change will use non-zero values, and fix them so the check passes.

* Add extra fields to remaining transaction value errors

* Swap the transparent value balance sign to match shielded value balances

This makes the signs of all the chain value pools consistent.

* Use a NonNegative constraint for transparent values

This fix:
* makes the type signature match the consensus rules
* avoids having to write code to handle negative values

* Allocate total generated transaction input value to outputs

If there isn't enough input value for an output, set it to zero.

Temporarily reduce all generated values to avoid overflow.
(We'll remove this workaround when we calculate chain value balances.)

* Consistently use ValueBalanceError for ValueBalances

* Make the value balance signs match the spec

And rename and document methods so their signs are clearer.

* Convert amount::Errors to specific pool ValueBalanceErrors

* Move some error changes to the next PR

* Add extra info to remaining transaction value errors (#2585)

* Distinguish between overflow and negative remaining transaction value errors

And make some error types cloneable.

* Add methods for updating chain value pools (#2586)

* Move amount::test to amount::tests:vectors

* Make ValueBalance traits more consistent with Amount

- implement Add and Sub variants with Result and Assign
- derive Hash

* Clarify some comments and expects

* Create ValueBalance update methods for blocks and transactions

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-09 14:22:26 -03:00
Alfredo Garcia 751185d4ec
Use the same value balance sign for transparent and shielded outputs (#2569)
* change signs

* make impl of Neg generic

* change implementation of Neg for Amount

Co-authored-by: teor <teor@riseup.net>
2021-08-08 22:41:34 +00:00
teor 8f4c3b09ea
Add test methods for modifying orchard shielded data and joinsplits (#2580)
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2021-08-07 13:23:32 +00:00
Conrado Gouvea fe989e0758
ZIP-221: Add Orchard support to history tree (#2531)
* Add Orchard support to HistoryTree

* Handle network upgrades in HistoryTree

* Add additional methods to save/load HistoryTree

* Apply suggestions from code review

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Clarification of Entry documentation

* Improvements from code review

* Add HistoryTree tests

* Improved test comments and variable names based on feedback from #2458 on similar test

* Update zebra-chain/src/history_tree.rs

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

* Use type aliases for V1 and V2 history trees

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-08-03 18:33:51 +00:00
Deirdre Connolly e719c46b1b
Track anchors and note commitment trees in zebra-state (#2458)
* Tidy chain Cargo.toml

* Organize imports

* Add method to get note commitments from all Actions in Orchard shielded data

* Add method to get note commitments from all JoinSplits in Sprout JoinSplitData

* Add Request and Response variants for awaiting anchors

* Add anchors and note commitment trees to finalized state db

* Add (From|Into)Disk impls for tree::Roots and stubs for NoteCommitmentTrees

* Track anchors and note commitment trees in Chain

Append note commitments to their trees when doing update_chain_state_with,
then use the resulting Sapling and Orchard roots to pass to history_tree, and add
new roots to the anchor sets.

* Handle errors when appending to note commitment trees

* Add comments explaining why note commitment are not removed from the tree in revert_chain_state_with

* Implementing note commitments in finalized state

* Finish serialization of Orchard tree; remove old tree when updating finalize state

* Add serialization and finalized state updates for Sprout and Sapling trees

* Partially handle trees in non-finalized state. Use Option for trees in Chain

* Rebuild trees when forking; change finalized state tree getters to not require height

* Pass empty trees to tests; use empty trees by default in Chain

* Also rebuild anchor sets when forking

* Use empty tree as default in finalized state tree getters (for now)

* Use HashMultiSet for anchors in order to make pop_root() work correctly

* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS

* Reduce DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES and MAX_PARTIAL_CHAIN_BLOCKS even more

* Apply suggestions from code review

* Add comments about order of note commitments and related methods/fields

* Don't use Option for trees

* Set DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES=1 and restore MAX_PARTIAL_CHAIN_BLOCKS

* Remove unneeded anchor set rebuilding in fork()

* Improve proptest formatting

* Add missing comparisons to eq_internal_state

* Renamed sprout::tree::NoteCommitmentTree::hash() to root()

* Improve comments

* Add asserts, add issues to TODOs

* Remove impl Default for Chain since it was only used by tests

* Improve documentation and assertions; add tree serialization tests

* Remove Sprout code, which will be moved to another branch

* Add todo! in Sprout tree append()

* Remove stub request, response *Anchor* handling for now

* Add test for validating Sapling note commitment tree using test blocks

* Increase database version (new columns added for note commitment trees and anchors)

* Update test to make sure the order of sapling_note_commitments() is being tested

* Improve comments and structure of the test

* Improve variable names again

* Rustfmt

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
Co-authored-by: Conrado P. L. Gouvea <conradoplg@gmail.com>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Co-authored-by: teor <teor@riseup.net>
2021-07-29 09:37:18 -04:00
teor 3d792f7195
Validate spends of transparent coinbase outputs (#2525)
* Validate transparent coinbase output maturity and shielding

- Add a CoinbaseSpendRestriction enum and Transaction method
- Validate transparent coinbase spends in non-finalized chains

* Don't use genesis created UTXOs for spends in generated block chains

* Refactor out a new_transaction_ordered_outputs function

* Add Transaction::outputs_mut for tests

* Generate valid transparent spends in arbitrary block chains

* When generating blocks, fixup the block contents, then the block hash

* Test that generated chains contain at least one transparent spend

* Make generated chains long enough for reliable tests

* Add transparent and shielded input and output methods to Transaction

* Split chain generation into 3 functions

* Test that unshielded and immature transparent coinbase spends fail

* Comment punctuation

* Clarify a comment

* Clarify probability calculation

* Test that shielded mature coinbase output spends succeed
2021-07-29 14:23:50 +10:00
Alfredo Garcia ee3c992ca6
Calculate the remaining value in the transparent transaction value pool (#2486)
* add value_balance methods to transparent and shielded

* add value_balance() to transaction

* check the remaining value consensus rule

* change error name

* fix doc and nitpick

* refactor value_balance() method for joinsplit

* changes to value_balance() of Inputs

* implement joinsplits() method(not working)

* remove created methods

* remove special case

* change return error in utilities

* move utils functions to transaction methods

* fix the docs

* simplify some code

* add constrains explicitly

* remove turbofish

* refactor some transaction methods

* fix value balance signs, add docs

* simplify some code

* avoid panic in consensus check

* add missing doc

* move remaining value balance check to the state

* make changes from the last review

Co-authored-by: teor <teor@riseup.net>
2021-07-29 13:49:36 +10:00
teor e6e03247ba
Reject UTXO double spends (#2511)
* Reject transparent output double-spends

Check that transparent spends use unspent outputs from:
* earlier transaction in the same block,
* earlier blocks in the parent non-finalized chain, or
* the finalized state.

* Fixup UTXOs in proptests

* Add a comment

* Clarify a consensus rule implementation

* Fix an incorrect comment

* Fix an incorrect error message

* Clarify a comment

* Document `unspent_utxos`

* Simplify the UTXO check

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Further simplify and fix the UTXO check

- split each error case into a separate check
- combine `contains` and `insert`
- add a missing check against the non-finalized unspent UTXOs
- rename arguments and edit error strings for clarity

* Share test methods between check test modules

* Make some chain fields available to tests

* Make error field names consistent with transparent::Input

* WIP: Add tests for UTXO double-spends

- accept output and spend in the same block
- accept output and spend in a later block
- reject output and double-spend all in the same block
- reject output then double-spend in a later block
- reject output, spend, then double-spend all in different blocks

* Use Extend rather than multiple pushes

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Use Extend for more pushes

* Limit the number of proptest cases, to speed up tests

* Test rejection of UTXOs that were never in the chain

* Test rejection of spends of later transactions in the same block

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-07-22 23:40:15 +00:00
Conrado Gouvea fdfa3cbdc6
Add ZIP-244 signature hash support (#2165)
* ZIP-244 sighash implementation using librustzcash

* ZIP-244: fix sighash test; add roundtrip test; update vectors

* Improvements from review; renamed sighash::Hash to SigHash
2021-07-07 08:27:10 +10:00
Conrado Gouvea dd645e7e0c
Add ZIP-0244 TxId Digest support (#2129)
* Add ZIP-0244 TxId Digest support

* Apply suggestions from code review

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-07-06 09:58:22 -03:00
Alfredo Garcia e4ab01dde0
ZIP-211: Validate Disabling Addition of New Value to the Sprout Value Pool (#2399)
* add disabled sprout pool check

* change method name

* change error name

* fix typo

* make the success test case in other tx than the coinbase

* use new `height` method instead of deriving `PartialOrd` in `NetworkUpgrade`

* move check of network upgrade into function, rename, docs

* increase test coverage

* fix comment
2021-07-02 09:03:34 +10:00
Alfredo Garcia c06cd19239
Update `has_inputs_and_outputs()` for new consensus rules (#2398)
* update the has_inputs_and_outputs() to new rules

* apply clippy suggestions

* add some TODOs
2021-06-29 08:28:49 +10:00
Alfredo Garcia 7638c43a7c
Move network_upgrade check into zebra-chain (#2354)
* move network_upgrade check into zebra-chain

* fix the errors

* rename function

* typo fix

* rename the check function

* make changes from last code review
2021-06-22 12:06:52 +10:00
teor 716fa23e86
Document shared to per-spend anchor conversion (#2363) 2021-06-22 11:41:35 +10:00