The ZTXOSelector type allows selection of previous Zcash
transaction outputs (both transparent outputs and shielded notes)
on the basis of either a (legacy) bare address, or for a
BIP-44 account.
We now use the empty array of pools to indicate that the default pools
should be used when providing a diversifier index parameter, instead of
needing a default sentinel value for the diversifier index parameter
(which was previously suggested as '*' which would have caused issues
for defining a consistent type for that parameter).
This adds an `allowRevealedAmounts` argument to z_sendmany. This
flag must be present to allow an amount-revealing cross-pool transfer
to be constructed.
This replaces the old implementation of asyncrpcoperation_sendmany
with one where all transaction construction is delegated to the
transaction builder. The capabilities of z_sendmany are somewhat
modified in the process:
* z_sendmany now permits sending funds from a Sprout address to
both transparent and Sapling addresses. PRIVACY NOTE: When
user sends a Sprout->Sapling transaction, the amount of the
transaction is publicly revealed.
* z_sendmany no longer supports transactions sending funds into
the Sprout pool, with the exception of change amounts when
sending from a Sprout address.
* When sending transparent coinbase funds to a set of shielded
addresses, the amount sent to recipients must fully consume
the input value and no change is permitted. This is a slightly
weaker constraint than was previously implemented; in the past,
only a single shielded recipient was allowed.
This adds a new `AddrSet` type for use in note retrieval
as a filter, in place of a heterogeneous list of `RawAddress`
values. `RawAddress` will complicate the handling of addresses
within the wallet after the addition of unified addresses,
because it does not contain transparent receiver types, and
if we retain this polymorphism it means a lot of invalid states
are represented in places we don't want them to be. It's better
to figure out what types of addresses we're working with as soon
as possible after parsing, and use monomorphic calls from there
on in.
We added support for the NU5 consensus rules in v4.5.0, which alters the
block header to contain a `hashBlockCommitments` value instead of the
chain history root. However, the output of `getblocktemplate` wasn't
returning this value; once NU5 activated, the `blockcommitmentshash`
field was being set to "null" (all-zeroes).
In v4.6.0 we added full NU5 support to `getblocktemplate`, by adding a
`defaultroots` field that gave default values for `hashBlockCommitments`
and the components required to derive it. However, in doing so we
introduced a regression in the (now-deprecated) legacy fields, where
prior to NU5 activation they contained nonsense.
This commit fixes the output of `getblocktemplate` to have the intended
semantics for all fields:
- The `blockcommitmentshash` and `authdataroot` fields in `defaultroots`
are now omitted from block templates for heights before NU5 activation.
- The legacy fields now always contain the default value to be placed
into the block header (regaining their previous semantics).
Co-authored-by: Larry Ruane <larry@z.cash>
This test currently fails with submitblock returning the error
"bad-heartwood-root-in-block".
Added authdigest to GBT coinbasetxn field because we can't obtain this
via getrawtransaction.
Co-authored-by: Jack Grigg <jack@z.cash>
This will ensure that miners can use the values returned by
getblocktemplate (in particular, the block commitment hash)
to submit a valid block using the submitblock RPC.
zcash/zcash:
The `getmininginfo` RPC now omits `currentblockize` and `currentblocktx`
when a block was never assembled via RPC on this node during its current
process instantiation. The relevant RPCs are `generate` (regtest only) and
`getblocktemplate`. Blocks are also assembled when running the internal
miner (`zcashd -gen=1`), after the node mines its first block.
(cherry picked from commit bitcoin/bitcoin@fa178a6385)
This test-only change allows python (rpc) tests to specify, for example,
that NU5 should be activated at height X, without having to specify all
the previous network upgrades. Previous upgrades can (and must) still be
specified if they activate at different block heights (than, in this
example, NU5). This makes tests easier to write (and read), especially
as the number of network upgrades increases over time.
Note that this change only affects zcashd behavior in regtest mode. For
the other network modes (testnet and mainnet), the activation heights
are hard-coded in chainparams.cpp.
Adds HD seed derivation from ZIP-339 mnemonic phrases.
This replaces random generation of transparent spending keys with BIP-44-compatible derivation from a mnemonic seed phrase, and replaces the random HD seed previously used for Sapling HD key derivation
with the same mnemonic seed. After this change, all new transparent and Sapling addresses
will be derived from a new randomly generated mnemonic seed, which is produced in such a way that
it is guaranteed to produce valid unified spending keys at account 0 for all address types.
Closes#5176.
Closes#2673.
After 4.5.2, all wallets will be populated with an emergency
recovery phrase, and all future addresses will be derived from
the associated seed. To prevent potential loss of funds, we
require that the user explicitly invoke the `walletconfirmbackup`
RPC method to verify that they have backed up this seed.
We aren't going to move to Clang 13 in a hotfix release. The other
dependencies passed their postponement re-evaluation date, but also
won't be updated just yet.
The "IsFromMe" logic was implemented in several places in the Bitcoin
Core wallet. We had correctly updated CWallet::IsFromMe(CTransaction)
(which was used in most places in the wallet) to check for shielded
notes being spent, but did not notice that CWalletTx::IsFromMe also
needed this check.
This bug has existed since before Zcash launched. It went unnoticed
because CWalletTx::IsFromMe was previously only called from code
used to either create purely-transparent transactions, or provide
informational output on non-critical RPC methods.
Closeszcash/zcash#5325.
- The patches `iostreams-106.patch` and `signals2-noise.patch` were
incorporated into Boost 1.75.
- The allocator access deprecation issue was fixed in Boost 1.76.
Closeszcash/zcash#4945.
This adds the double-hash message variant. The extra hash field is set
to null for block message types, and to all-ones for MSG_TX (to match
the legacy authHash value used for pre-v5 transactions in the Merkle
tree).
I tested the NU5 components of this PR by locally setting the protocol
version to 170014. I forgot to check that without that override, the
test would skip the NU5 checks. The reason it defaulted to NU5 is that
the test was reading the client version 4040150, which is indeed not
less than the NU5 protocol version ^_^;;
Extend P2P test framework to make it possible to expect reject
codes for transactions and blocks.
(cherry picked from commit bitcoin/bitcoin@20411903d7)
ZIP 239 preparations 2
Cherry-picked from the following upstream PRs:
- bitcoin/bitcoin#6722
- Only the ancillary commits, not the mempool limiting commits (we have our own).
- bitcoin/bitcoin#6898
- Only the first three commits (we'll cherry-pick the main content later).
- bitcoin/bitcoin#7840
Also remove the RPC deprecation tests for accounts, and make one small
change to another wallet test that relies on account behaviour.
(cherry picked from commit f0dc850bf698f7377797d7d68365d4fc79b0221c)
Previously Bitcoin would send 1/4 of transactions out to all peers
instantly. This causes high overhead because it makes >80% of
INVs size 1. Doing so harms privacy, because it limits the
amount of source obscurity a transaction can receive.
These randomized broadcasts also disobeyed transaction dependencies
and required use of the orphan pool. Because the orphan pool is
so small this leads to poor propagation for dependent transactions.
When the bypass wasn't in effect, transactions were sent in the
order they were received. This avoided creating orphans but
undermines privacy fairly significantly.
This commit:
Eliminates the bypass. The bypass is replaced by halving the
average delay for outbound peers.
Sorts candidate transactions for INV by their topological
depth then by their feerate (then hash); removing the
information leakage and providing priority service to
higher fee transactions.
Limits the amount of transactions sent in a single INV to
7tx/sec (and twice that for outbound); this limits the
harm of low fee transaction floods, gives faster relay
service to higher fee transactions. The 7 sounds lower
than it really is because received advertisements need
not be sent, and because the aggregate rate is multipled
by the number of peers.
(cherry picked from commit f2d3ba73860e875972738d1da1507124d0971ae5)
Zcash: Candidate transactions for INV are not sorted by their
topological depth because we haven't backported bitcoin/bitcoin#6654.
ZIP 239 preparations 1
This is the first of several backports to prepare for ZIP 239. The primary
change is altering `mapRelay` to store `CTransaction`s, which we need
because ZIP 239 requires changing `Inv` messages based on transaction
versions. The other changes are mainly for conflict removal but are also
independently useful.
Backports the following upstream PRs:
- bitcoin/bitcoin#6889
- bitcoin/bitcoin#7125
- bitcoin/bitcoin#7862
- bitcoin/bitcoin#7877
The recent changes to mempool inv logic mean that nodes are much less
likely to immediately return an `inv` message in response to a `mempool`
message. The `p2p_txexpiringsoon` RPC test was relying on the prior
behaviour.
`TestNode.sync_with_ping` now takes an optional `waiting_for` closure
that allows the caller to require that a specific message kind is
received prior to the timeout.
This is the second release in a row where LLVM has cut a X.0.1 for
everything except Darwin, so I've adjusted its URLs and paths on the
assumption this will continue.
Bitcoin Core doesn't actually use tags for managing versions of their
forked dependencies, so we should separately rework this logic for all
of the subtree-managed dependencies. But this at least prevents false
positives.
Split wallet.py
Split some of the tests from `wallet.py` into its own files to make it easier to debug.
Some of the tests moved where depending(specially in balance amounts) on previous code. As standalone, some of the hardcoded balances needed some changes. No further modifications to tests are done in this PR.
We can split more but i think is a good start.
If this is merged it will close https://github.com/zcash/zcash/issues/4917
-----------------------
Please ensure this checklist is followed for any pull requests for this repo. This checklist must be checked by both the PR creator and by anyone who reviews the PR.
* [ ] Relevant documentation for this PR has to be completed and reviewed by @mdr0id before the PR can be merged
* [ ] A test plan for the PR must be documented in the PR notes and included in the test plan for the next regular release
As a note, all buildbot tests need to be passing and all appropriate code reviews need to be done before this PR can be merged
qa/zcash/updatecheck.py: remove dead code; print instructions to run `cargo outdated` and `cargo update`
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Reduce default fee to 1000 zatoshis
Per ZIP 313. This also ensures that transactions that pay the default fee will always be relayed, and not rate-limited.
This removes the paches iostreams-106.patch and signals2-noise.patch
which have been incorporated into boost 1.75. Also, this further
postpones updates to native_clank, libcxx and native_ccache.
A few miscellaneous improvements to rpc-tests.py command line arguments:
- make all arguments start with double dash for consistency
- improve help text and output
- add nozmq argument to explicitly exclude the ZMQ tests
- change 'parallel' to 'jobs'
- Link pull-tester/rpc-tests.py to the build dir
- Add the build-dir's config to the python path so that tests can find it
- The tests themselves are in srcdir
- Clean up __pycache__ in 'make clean'
Previously these functions would infinitely loop if sync failed;
now they have a default timeout of 60 seconds, after which an
AssertionError is raised.
sync_blocks() has also been improved and now compares the tip
hash of each node, rather than just using block count.
Zcash: Kept block count check for a couple of tests where we use it.
If ZMQ is enabled, check whether it's installed before running ZMQ tests.
If it isn't, disable ZMQ and print a warning.
Also add dependency info to test docs, so users know ZMQ is required
before running tests, and so they know how to install it.
When following the build instructions before this change then trying
to run the RPC tests, a unix user would get an error when python
tried to import zmq.
There may be other dependencies that should be added to the docs,
particularly ones for non-unix systems. This is the only unlisted
dependency I encountered using linux.
In rpc-tests.py, don't override BITCOIND and BITCOINCLI if they're
already set. Makes it possible to run the tests with either another tree
or the GUI.
Thanks to @MarcoFalke @dexX7 @laanwj for review.
Zcash: Just the --coverage flag for rpc-tests.py, as we backported the
rest of the coverage backend in zcash/zcash#4411.
1) created rpc-tests.py
2) deleted rpc-tests.sh
3) travis.yml points to rpc-tests.py
4) Modified Makefile.am
5) Updated README.md
6) Added tests_config.py and deleted tests-config.sh
7) Modified configure.ac with script to set correct path in tests_config.py
Zcash: Migrated our test list over, and other necessary modifications.
The UI changes were not migrated.
Since the introduction of the ScriptForMining callback, the mining
functions (setgenerate and generate) crash with an assertion failure
(due to a NULL pointer script returned) if the keypool is empty. Fix
this by giving a proper error.
Zcash: Adapted to our MinerAddress type.
Co-authored-by: Jack Grigg <jack@electriccoin.co>
test: Use default shielded address in RPC tests where the type is irrelevant
This means we are running these tests against the recommended functionality,
and that these RPC tests will now use the faster Sapling addresses.
Improving asyncOp_sendmany redundancies (first round)
Work over async sendmany operation flow. Specifically on the utxo finding, utxo selection, total amount calculation and the tx inputs appending process. Removed several unneeded for loops over the entire utxo set, re structuring and cleaning the flow.
These are text files but their encoding does not depend on the locale.
Not all of them require utf8 but it is better to fix it at something
to remove potential unpredictability.
This is necessary on FreeBSD where no locale is set by default,
and apparently Python defaults not only the terminal encoding to the locale
but that of every text file. So without LOCALE environment it defaults text
file encoding to ASCII. This causes problems with e.g. `bitcoin.conf`.
Luckily the locale doesn't affect the default encoding for str.encode() and
bytes.decode() on Python 3, so this is the only change necessary.
allow getaddressutxos if -lightwalletd
The wallets need to be able to retrieve, via lightwalletd, the UTXO set for a given address, see https://github.com/zcash/lightwalletd/issues/312. The `getaddressutxos` rpc that was added for Insight Explorer is perfect for this, but this rpc is disabled when only `-lightwalletd` is enabled (that is, it requires `-insightexplorer`, which is a superset of `-lightwalletd`). However, the `DB_ADDRESSUNSPENTINDEX` index that supports this rpc is present when only `-lightwalletd` is enabled. This was probably a (fortuitous, as it turns out) oversight. So all that's required is to allow this rpc when `-lightwalletd` is enabled.
We don't currently have a need for newer releases, and will likely be
pinning Rust for longer periods of time once we are also pinning Clang.
Rust releases occur every six weeks, so we can pre-emptively postpone
releases through the end of this year.
- The old patch is no longer necessary because of this upstream fix:
https://github.com/boostorg/build/pull/560
- Boost 1.72 removed a <deque> from an include, which exposed a missing
include in src/httpserver.cpp.
- Boost 1.73 moved function placeholders into the boost::placeholders
namespace.
- The new patch is a fix from just after Boost 1.74 was released, fixing
a warning that was missed.
Small httpserver.cpp backports
Also includes a change to the `uiInterface.NotifyBlockTip` signal API.
These remove merge conflicts from subsequent backports for `sync.h`.
Cherry-picked from the following upstream PRs:
- bitcoin/bitcoin#6859
- bitcoin/bitcoin#7112
- Only the non-QT changes.
- bitcoin/bitcoin#7966
- bitcoin/bitcoin#8421
- We already backported the second commit in zcash/zcash#2555
Address some performance regressions
@str4d identified and fixed some performance regressions in our elliptic curve and proving crates, and we also changed to complete formulas in `bls12_381` to get some better performance in some cases. The result:
```
v3.1.0
"runningtime": 0.59883
before upgrading rust:
"runningtime": 0.823932
after upgrading rust:
"runningtime": 0.83004
after upgrading rust packages:
"runningtime": 0.763065
```
which gets us some of the way back to where we were.
A few "a->an" and "an->a".
"Shows, if the supplied default SOCKS5 proxy" -> "Shows if the supplied default SOCKS5 proxy". Change made on 3 occurrences.
"without fully understanding the ramification of a command" -> "without fully understanding the ramifications of a command".
Removed duplicate words such as "the the".
Zcash: Only the changes to files and code that we have.
wallet: Add ANY_TADDR special string to z_sendmany
When using this special string as the from address, non-coinbase UTXOs
from any transparent addresses within the wallet will be used to fund the
transaction. Change outputs will be sent to a new transparent address,
as with any other spend of transparent funds.
Closeszcash/zcash#3640.
depends: Switch to `cargo vendor` for Rust dependencies
When we first integrated Rust into our build system, we had two
limitations:
- We were building the `librustzcash` FFI library as a dependency, and
therefore needed access to its crate dependencies in the depends
system.
- Gitian builds happen offline, so we needed to fetch any crate
dependencies ahead of time, and then configure cargo to use these in
an offline environment.
At the time, `cargo` already had support for "Source Replacement", but
there was no easy way to package the dependencies in the necessary way.
What we implemented was effectively the `cargo-vendor` tool, built using
Makefiles. A noticeable downside was that we were pinning dependencies
twice: once in the `Cargo.lock` for the FFI library, and again in our
depends system.
Since then, `cargo-vendor` has been upstreamed into `cargo` itself, and
we have moved `librustzcash` into this repository. We can therefore use
`cargo vendor` directly from our pinned Rust compiler to fetch the
dependencies, and rely on our local `Cargo.lock` to pin the specific
crates we are relying on.
add shielded balance to getwalletinfo
Closeszcash/zcash#3939
It is based on the definition that unconfirmed balance has 0 confirmations; anything else is regular balance.
Implementation:
1. `getBalanceZaddr` uses one version of `GetFilteredNotes`, we want to use the other version that allow us to get balances inside min and max confirmations.
2. `shielded_unconfirmed_balance`, and `shielded_balance` are obtained by calling `getBalanceZaddr` with different min and max confirmations according to the definitions from above.
When using this special string as the from address, UTXOs from any
transparent addresses within the wallet will be used to fund the
transaction. Change outputs will be sent to a new transparent address,
as with any other spend of transparent funds.
Closeszcash/zcash#3640.
Use the Rust tracing crate for C++ logging
This PR swaps in the `tracing` crate (via FFI) for logging to either standard
output or `debug.log`. It transparently maps all existing `LogPrintf` and
`LogPrint` invocations to info-level `tracing` events, and passes through
correct file and line information. `error` invocations are mapped to error-level
`tracing` events, currently without line information (due to the way that
`error` is used in the codebase; swapping individual callsites to the new
`LogError` macro will provide that information).
The end-goal for this change is that we don't need to make any disruptive
changes to the codebase, but we can start to leverage `tracing`-specific
functionality where we want to, such as providing extra fields on certain log
lines (that can be filtered for), adding spans to record the flow of execution
through `zcashd`, and logging within C++ and Rust simultaneously. Support
for extra fields on spans and events will be added in a subsequent PR.
The `-debug` config options are converted at launch into their corresponding
directives for tracing's `EnvFilter`. The new `setlogfilter` RPC method allows
this filter to be reloaded dynamically. The syntax is documented in the
`setlogfilter` help text, as well as here:
https://docs.rs/tracing-subscriber/0.2.7/tracing_subscriber/filter/struct.EnvFilter.html#directives
When `-printtoconsole` is specified, the output now includes timestamps and
ANSI encoding :)
A new safe_depends() checker is added, which allows the '.' character on
top of the existing safe() characters.
The time crate is postponed to 2021-02-01, by which time hopefully the
chrono crate will have figured out what it is doing with that dependency.
flush witness cache (SetBestChain()) on clean shutdown
Closes#4596, follow-on to #4573. In addition to flushing witness data on shutdown, fix the RPC test that was preventing this change from being part of #4573.
Flush witness data to disk only when it's consistent
Closes#4301. Running this PR's code will not repair a data directory that has been affected by this problem; that requires starting zcashd with the `-rescan` or `-reindex` options.
This makes the test more deterministic. When the nodes are split, and
partition B (node 1) mines a joinsplit transaction, that block only
sometimes appears on the blockchain after the nodes are all reconnected.
ZIP212 implementation
Closes#4557.
(description by @ebfull, taken from #4575)
* The `SaplingNote` structure has a new enum called `zip212Enabled`. This
member is private and reflects whether the note was or is being created
using the derivation method of ZIP 212 (i.e., `BeforeZip212` or `AfterZip212`).
* The `SaplingNotePlaintext` structure has a new unsigned char member
`leadbyte`. This member is private and contains the leading byte of the
plaintext (e.g. `0x01`, `0x02`).
* The serialization of `SaplingNotePlaintext` sets `zip212Enabled` to
`BeforeZip212` iff the serialized note plaintext version is not `0x01`.
* The `r`/`rcm` fields have been removed and replaced with a private field
`rseed`. `SaplingNote` and `SaplingNotePlaintext` now have a helper method
`rcm()` which returns the `rcm` either by deriving it with `rseed`
(if `zip212Enabled` is `AfterZip212`) or returning `rseed` by interpreting
`rseed` as `rcm`.
* All the methods of obtaining a `SaplingNote` account for these changes:
- The `SaplingNote` constructor that is used by e.g. the transaction builder,
and internally samples random `rcm`, now takes a `zip212Enabled` argument
to decide whether to sample `rcm` the "old" way or the "new" way.
- The bare constructor for `SaplingNote` is removed.
- The other constructor which takes the raw contents of the note is only used
in tests or in `Note.cpp`, but now also takes a `zip212Enabled` argument.
- The other way of obtaining a note, by calling `SaplingNotePlaintext::note()`,
has been adjusted.
* The `SaplingNotePlaintext` class now has an `generate_or_derive_esk()` method
that either samples a random `esk` or derives it using the local `rseed`
depending on the value of `leadbyte`.
* The encryption routine is modified to consult `generate_or_derive_esk()` and
provide it to the note encryption object.
* The note encryption objects now take an optional `esk` as input and otherwise
sample a random `esk` internally. This API functionality is preserved to allow
for testing.
* The `SaplingNotePlaintext` decryption routines are modified:
- The out and enc decryption routines now check that `epk` is consistent with
the derived `esk`.
- The out decryption routine for plaintexts also checks that `esk` is
consistent with what is derived by the note.
* The miner and transaction builder consult the activation of Canopy when
creating `SaplingNote`s.
* The consensus rules are modified so that shielded outputs (miner rewards)
must have `v2` note plaintexts after Canopy has activated.
[ZIP 211] Disabling Addition of New Value to the Sprout Value Pool
Disables Sprout outputs after NU4 by checking for nonzero `vpub_old` in transactions after NU4 activation height.
Adds gtests to check expected behaviour before and after NU4 activation height.
edit:
Also modifies `z_` methods in `rpcwallet`, and adds a matching RPC test.
Implements [ZIP 211](https://zips.z.cash/zip-0211), closes#4479
Add funding streams to consensus parameters.
Add funding stream payments to coinbase txns generated by the miner.
* Reduce valueBalance for shielded outputs to funding streams.
* Ensure we produce binding signatures in any case where shielded
outputs go to either a funding stream or the miner.
Use the cached consensusBranchId in DisconnectBlock
If a node is started with a set of network upgrades that don't match the
serialized chain (such as when we implement NU rollbacks on testnet),
RewindBlockIndex will disconnect each block in the chain until it
reaches the most recent block that agrees with the node's set of network
upgrades. However, the blocks themselves should be disconnected using
the consensus branch ID that they were connected with, which is
persisted alongside the chain and reconstructed in LoadBlockIndex.
Kris Nuttycombe
str4d
Jack Grigg
Charlie O'Keefe
zancas
Larry Ruane
Dimitris Apostolou
Larry Ruane
MarcoFalke
Kris Nuttycombe
mdr0id
Wladimir J. van der Laan
Shaul Kfir
Homu
John Newbery
Gregory Maxwell
Pieter Wuille
str4d
Steven Smith
therealyingtong
Jonas Schnelli
Suhas Daftuar
Jonas Schnelli
Taylor Hornby
Alfredo Garcia
Daira Hopwood
Josh Ellithorpe
fanquake
Alex Morcos
isle2983
whythat
Cory Fields
Suhas Daftuar
Elliot Olds
James O'Beirne
Jeff Garzik
ptschip
Casey Rodarmor
Daniel Kraft
Jesse Cohen
furszy
Sean Bowe
Dimitris Tsapakidis
zebambam
therealyingtong
Kris Nuttycombe
therealyingtong
Rod Vagg
Anthony Towns
Eirik Ogilvie-Wigley