Commit Graph

20 Commits

Author SHA1 Message Date
teor e17542cfb8
fix(log): Stop logging potentially sensitive user information from unmined transactions (#6616)
* Stop deriving Debug impl

* Change formatting for unmined transactions to keep user info confidential

* Fix derives
2023-05-16 23:55:45 +00:00
Alfredo Garcia 8075d616fb
feat(mempool): add ZIP-317 rules to mempool (#6556)
* add ZIP-317 rules to mempool

* fix some docs

* rustfmt

* fix import

* typo

* fix tests

* fix tests 2

* fix tests 3

* fix tests 4

* fix tests 5

* move constant

* fix constant for feature

* document/quote zip rules

* add Minimum Fee Rate rule

* change(mempool): Refactor the ZIP-317 minimum fee rate calculation to use usize (#6585)

* Refactor the minimum fee rate calculation to use usize

* Check for overflow if constants change

* remove 1 rule check, fix docs

---------

Co-authored-by: teor <teor@riseup.net>
2023-05-02 00:13:33 +00:00
teor f2133c9fff
Increase ZIP-401 mempool cost thresholds for Orchard transactions (#6521)
* Increase ZIP-401 mempool cost thresholds for Orchard transactions

* Copy documentation from ZIP-401
2023-04-18 10:11:58 +00:00
teor 166526a088
feat(ui): Add a terminal-based progress bar to Zebra (#6235)
* Implement Display and to_string() for NetworkUpgrade

* Add a progress-bar feature to zebrad

* Add the progress bar writer to the tracing component

* Add a block progress bar transmitter

* Correctly shut down the progress bar, and shut it down on an interrupt

* Make it clearer that the progress task never exits

* Add a config for writing logs to a file

* Add a progress-bar feature to zebra-network

* Add a progress bar for the address book size

* Add progress bars for never attempted and failed peers

* Add an optional limit and label to connection counters

* Add open connection progress bars

* Improve CheckpointList API and CheckpointVerifier debugging

* Add checkpoint index and checkpoint queue progress bars

* Security: Limit the number of non-finalized chains tracked by Zebra

* Make some NonFinalizedState methods available with proptest-impl

* Add a non-finalized chain count progress bar

* Track the last fork height for newly forked chains

* Add a should_count_metrics to Chain

* Add a display method for PartialCumulativeWork

* Add a progress bar for each chain fork

* Add a NonFinalizedState::disable_metrics() method and switch to using it

* Move metrics out of Chain because we can't update Arc<Chain>

* Fix: consistently use best chain order when searching chains

* Track Chain progress bars in NonFinalizedState

* Display work as bits, not a multiple of the target difficulty

* Handle negative fork lengths by reporting "No fork"

* Correctly disable unused fork bars

* clippy: rewrite using `match _.cmp(_) { ... }`

* Initial mempool progress bar implementation

* Update Cargo.lock

* Add the actual transaction size as a description to the cost bar

* Only show mempool progress bars after first activation

* Add queued and rejected mempool progress bars

* Clarify cost note is actual size

* Add tracing.log_file config and progress-bar feature to zebrad docs

* Derive Clone for Chain

* Upgrade to howudoin 0.1.2 and remove some bug workarounds

* Directly call the debug formatter to Display a Network

Co-authored-by: Arya <aryasolhi@gmail.com>

* Rename the address count metric to num_addresses

Co-authored-by: Arya <aryasolhi@gmail.com>

* Simplify reverse checkpoint lookup

Co-authored-by: Arya <aryasolhi@gmail.com>

* Simplify progress bar shutdown code

Co-authored-by: Arya <aryasolhi@gmail.com>

* Remove unused MIN_TRANSPARENT_TX_MEMPOOL_SIZE

* Document that the progress task runs forever

* Fix progress log formatting

* If progress-bar is on, log to a file by default

* Create missing directories for log files

* Add file security docs for running Zebra with elevated permissions

* Document automatic log file, spell progress-bar correctly

---------

Co-authored-by: Arya <aryasolhi@gmail.com>
2023-04-13 08:42:17 +00:00
teor cb1045ae5f
change(mempool): Re-verify mempool transactions after a chain fork, rather than re-downloading them all (#5841)
* Move Drop from mempool::ActiveState to mempool::Downloads, to avoid bugs

* Re-verify mempool transactions after a fork

And add a marker struct for mempool download cancellation.

* Update README based on recent mitigations for some issues, tidy format

* Make mempool proptests easier to debug

* Make UnminedTx Display text much smaller

* Update tests for mempool transaction re-verification after forks

* Retry all stored and pending transactions

* Fix a test to check for mempool reset retries
2022-12-12 23:19:45 +00:00
teor bb5f9347ea
change(rpc): Update ZIP-317 transaction selection algorithm (#5776)
* Update ZIP-317 implementation for unpaid actions

* Split shared transaction choose and check into its own function

* Fix an incorrect address error message

* Simplify code, expand docs

* Require docs for getblocktemplate RPC types

* Account for the coinbase transaction in the transaction selection limits

* Fix a broken doc link, update comments, tidy imports

* Fix comment typos

* Use the actual block height rather than Height::MAX for the fake coinbase

* Use a 1 zat fee rather than 0, just in case someone gets clever and skips zero outputs
2022-12-08 04:19:12 +00:00
teor d778caebb8
change(rpc): Select getblocktemplate RPC transactions according to ZIP-317 (#5724)
* Split the conventional fee check into its own method

This will be used for block production and relaying

* Move getblocktemplate transaction selection into a new zip317 module

* Add a block_production_fee_weight field to VerifiedUnminedTx

* Add a custom Zebra minimum transaction weight for block production

* Implement ZIP-317 transaction selection for block production

* Split weighted index setup into its own function

* Split picking a transaction into its own function
2022-12-01 21:57:22 +00:00
teor 63124ba962
change(mempool): Evict transactions from the mempool using the ZIP-317 conventional fee (#5703)
* Add a ZIP-317 conventional fee module

* Add a conventional fee calculation stub, and use it for mempool size limiting

* Just return a usize from zcash_serialized_size(), removing the unused Result

* Add ZIP-317 constants

* Calculate the ZIP-317 conventional fee

* Update tests

* Add a CHANGELOG entry

* Fix a comment typo

Co-authored-by: Daira Hopwood <daira@jacaranda.org>

* Fix some missing words in a comment

Co-authored-by: Arya <aryasolhi@gmail.com>

Co-authored-by: Daira Hopwood <daira@jacaranda.org>
Co-authored-by: Arya <aryasolhi@gmail.com>
2022-11-24 01:27:35 +00:00
teor 71f5e63e64
3. change(rpc): Add fee and sigops fields to getblocktemplate transactions (#5508)
* Add a legacy_sigop_count field to VerifiedUnminedTx

* Add conversions from Vec<VerifiedUnminedTx> to block header roots

* Add fee and sigops field to block template transactions

* Fix up mempool request names

* Increase existing snapshot test coverage

* Document a new method parameter

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-11-03 17:03:41 +00:00
teor 142411508b
2. change(rpc): Add some transaction fields to the `getblocktemplate` RPC (#5496)
* Add documentation for the getblocktemplate RPC

* Add a new mempool::Request::Transactions

* Add conversions from Vec<UnminedTx> to merkle::Root and AuthDataRoot

* Fill in the merkle root and auth data root fields

* Delete the Coinbase type, it's the same as Transaction

* Fill in some other existing types

* Add Hex serialization support to some zebra-chain types

* Add TransactionTemplate fields and fill some in

* Fix test hangs by spawning async tasks

* Add temporary workaround for no transactions in the block

* Encode hashes and roots as hex

* Update RPC snapshots

* Add a missing Request::Transactions handler

* Fix doc warnings

* Fix fee serialization

* Update snapshots for serialization changes

* Add a missing Cargo.lock change

* Change depends type

* Remove duplicate feature entry

* Document the new RPC feature

* Fix a comment typo

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Update default roots docs

* Fix comment typo

* Fix a comment typo

Co-authored-by: Arya <aryasolhi@gmail.com>

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Co-authored-by: Arya <aryasolhi@gmail.com>
2022-11-03 03:25:01 +00:00
teor 42ef8846b3
fix(clippy): resolve or disable new nightly clippy lints (#4739)
* Fix new dead_code lints in generated pin-project code

* Fix and ignore new needless_borrow lints
2022-07-05 11:04:47 +10:00
Marek b8b35f8da9
fix(doc): Fix various doc warnings, part 1 (#4514)
* Fix the syntax of links in comments

* Fix a mistake in the docs

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>

* Remove unnecessary angle brackets from a link

* Revert the changes for links that serve as references

* Revert "Revert the changes for links that serve as references"

This reverts commit 8b091aa9fa.

* Remove `<` `>` from links that serve as references

This reverts commit 046ef25620.

* Don't use `<` `>` in normal comments

* Don't use `<` `>` for normal comments

* Revert changes for comments starting with `//`

* Fix some warnings produced by `cargo doc`

* Fix some rustdoc warnings

* Fix some warnings

* Refactor some changes

Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
2022-06-02 15:07:35 +00:00
Deirdre Connolly 0381c2347b
ZIP-401: weighted random mempool eviction (#2889)
* ZIP-401 weighted random mempool eviction

* rename zcash.mempool.total_cost.bytes to zcash.mempool.cost.bytes

Co-authored-by: teor <teor@riseup.net>

* Remove duplicated lines

* Add cost() method to UnminedTx

Update serialization failure messages

* More docs quoting ZIP-401 rules

* Change mempool::Storage::new() to handle Copy-less HashMap, HashSet

* mempool: tidy cost types and evict_one()

* More consensus rule docs

* Refactor calculating mempool costs for Unmined transactions

* Add a note on asympotic performance of calculating weights of txs in mempool

* Bump test mempool / storage config to avoid weighted random cost limits

* Use mempool tx_cost_limit = u64::MAX for some tests

* Remove failing tests for now

* Allow(clippy::field-reassign-with-default) because of a move on a type that doesn't impl Copy

* Fix mistaken doctest formatting

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

* Increase test timeout for Windows builds

Co-authored-by: teor <teor@riseup.net>
Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
2021-10-27 00:21:19 +00:00
teor 2d129414e0
Store the transaction fee in the mempool storage (#2885)
* Create a new VerifiedUnminedTx containing the miner fee

* Use VerifiedUnminedTx in mempool verification responses

And do a bunch of other cleanups.

* Use VerifiedUnminedTx in mempool download and verifier

* Use VerifiedUnminedTx in mempool storage and verified set

* Impl Display for VerifiedUnminedTx, and some convenience methods

* Use VerifiedUnminedTx in existing tests
2021-10-18 11:24:37 +10:00
teor 2f0926a8e4
Stop ignoring the mempool conflicting transaction reject list size limit (#2855)
* Limit the size of rejection lists when there is a spend conflict

Previously, `insert` would return early with an error,
and skip limiting the rejection list sizes.

* Use prop_assert macros in proptests, rather than assert
2021-10-12 10:35:50 +10:00
Alfredo Garcia f1718f5c92
Add `zcash_serialized_size()` to `ZcashSerialize` trait (#2824)
* add a zcash_serialized_size()

* add a size field to `UnminedTx`

* refactor zcash_serialized_size() to don't allocate RAM

* improve performance

Co-authored-by: teor <teor@riseup.net>

* clippy

Co-authored-by: teor <teor@riseup.net>
2021-10-06 22:40:11 +00:00
Conrado Gouvea 1ccb2de7c7
Add transaction downloader and verifier (#2679)
* Add transaction downloader

* Changed mempool downloader to be like inbound

* Verifier working (logs result)

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* Fix coinbase check for mempool, improve is_coinbase() docs

* Change other downloads.rs docs to reflect the mempool downloads.rs changes

* Change TIMEOUTs to downloads.rs; add docs

* Renamed is_coinbase() to has_valid_coinbase_transaction_inputs() and contains_coinbase_input() to has_any_coinbase_inputs(); reorder checks

* Validate network upgrade for V4 transactions; check before computing sighash (for V5 too)

* Add block_ prefix to downloads and verifier

* Update zebra-consensus/src/transaction.rs

Co-authored-by: teor <teor@riseup.net>

* Add consensus doc; add more Block prefixes

Co-authored-by: teor <teor@riseup.net>
2021-09-02 00:06:20 +00:00
teor 2ed6679069
Use unmined types for transaction verifier mempool requests and responses (#2666)
* Add `Arc<Transaction>` conversions for Transaction IDs

* Use UnminedTxId as the transaction verifier response type

* Use UnminedTx in transaction verifier mempool requests

* Refactor is_mempool into a transaction verifier request method

* Order derives in alphabetical order

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2021-08-25 21:02:47 +00:00
teor c608260256
Support witnessed transaction IDs in zebra-network requests and responses (#2638)
* Rename internal network requests for wide transaction IDs

fastmod TransactionsByHash TransactionsById zebra*
fastmod AdvertiseTransactions AdvertiseTransactionIds zebra*
fastmod MempoolTransactions MempoolTransactionIds zebra*
fastmod TransactionHashes TransactionIds zebra*

* Update network transaction request/response comments

* Rename a transaction hash method for wide transaction IDs

fastmod transaction_hashes transaction_ids zebra-network

* Add UnminedTxId methods and conversions for InventoryHash

* Map WtxIds to unmined transaction network messages

Also, use UnminedTxId and UnminedTx in:
* Zebra's internal request and response format, and
* external Zcash network protocol messages.

* Enable WtxId mempool inventory tracking for peers

* Further clarify transaction IDs

* Use Witnessed rather than Wide for transaction IDs

And rename narrow to legacy when it only applies to v1-v4 transactions.
Otherwise, rename it to mined ID.

* Rename a missed binding
* Remove an incorrectly named binding

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
2021-08-18 22:55:24 +00:00
teor 84c5f6189d
Create types for unmined transactions and their IDs (#2634)
* Create new types for unmined transactions and their IDs

* Add accessor methods for the parts of an unmined transaction ID
2021-08-18 15:52:42 +10:00