Flush witness data to disk only when it's consistent
Closes#4301. Running this PR's code will not repair a data directory that has been affected by this problem; that requires starting zcashd with the `-rescan` or `-reindex` options.
ZIP212 implementation
Closes#4557.
(description by @ebfull, taken from #4575)
* The `SaplingNote` structure has a new enum called `zip212Enabled`. This
member is private and reflects whether the note was or is being created
using the derivation method of ZIP 212 (i.e., `BeforeZip212` or `AfterZip212`).
* The `SaplingNotePlaintext` structure has a new unsigned char member
`leadbyte`. This member is private and contains the leading byte of the
plaintext (e.g. `0x01`, `0x02`).
* The serialization of `SaplingNotePlaintext` sets `zip212Enabled` to
`BeforeZip212` iff the serialized note plaintext version is not `0x01`.
* The `r`/`rcm` fields have been removed and replaced with a private field
`rseed`. `SaplingNote` and `SaplingNotePlaintext` now have a helper method
`rcm()` which returns the `rcm` either by deriving it with `rseed`
(if `zip212Enabled` is `AfterZip212`) or returning `rseed` by interpreting
`rseed` as `rcm`.
* All the methods of obtaining a `SaplingNote` account for these changes:
- The `SaplingNote` constructor that is used by e.g. the transaction builder,
and internally samples random `rcm`, now takes a `zip212Enabled` argument
to decide whether to sample `rcm` the "old" way or the "new" way.
- The bare constructor for `SaplingNote` is removed.
- The other constructor which takes the raw contents of the note is only used
in tests or in `Note.cpp`, but now also takes a `zip212Enabled` argument.
- The other way of obtaining a note, by calling `SaplingNotePlaintext::note()`,
has been adjusted.
* The `SaplingNotePlaintext` class now has an `generate_or_derive_esk()` method
that either samples a random `esk` or derives it using the local `rseed`
depending on the value of `leadbyte`.
* The encryption routine is modified to consult `generate_or_derive_esk()` and
provide it to the note encryption object.
* The note encryption objects now take an optional `esk` as input and otherwise
sample a random `esk` internally. This API functionality is preserved to allow
for testing.
* The `SaplingNotePlaintext` decryption routines are modified:
- The out and enc decryption routines now check that `epk` is consistent with
the derived `esk`.
- The out decryption routine for plaintexts also checks that `esk` is
consistent with what is derived by the note.
* The miner and transaction builder consult the activation of Canopy when
creating `SaplingNote`s.
* The consensus rules are modified so that shielded outputs (miner rewards)
must have `v2` note plaintexts after Canopy has activated.
This fixes wallet_anchorfork.py CI failure, but a separate PR
will restore flushing witness data on shutdown while also
fixing DecrementNoteWitnesses() to not assert when
nd->witnessHeight < indexHeight, which can happen when the
node reorgs upon restart (which this test causes to happen).
It needs to be closer to the root of our dependency tree, so that it can
depend on the transaction format. The libzcash compilation unit is
further from the dependency tree root than the transaction format.
We don't support making pre-Sapling JoinSplit proofs, and we load the
parameters for post-Sapling JoinSplit proofs at proving time, so there
is no need for a global ZCJoinSplit to be passed through the APIs.
We only relied on success being 0 and our code was otherwise agnostic to the
actual return code in the event of failed signature verification, but this
change keeps the API consistent.
[ZIP 211] Disabling Addition of New Value to the Sprout Value Pool
Disables Sprout outputs after NU4 by checking for nonzero `vpub_old` in transactions after NU4 activation height.
Adds gtests to check expected behaviour before and after NU4 activation height.
edit:
Also modifies `z_` methods in `rpcwallet`, and adds a matching RPC test.
Implements [ZIP 211](https://zips.z.cash/zip-0211), closes#4479
Add funding streams to consensus parameters.
Add funding stream payments to coinbase txns generated by the miner.
* Reduce valueBalance for shielded outputs to funding streams.
* Ensure we produce binding signatures in any case where shielded
outputs go to either a funding stream or the miner.
SaplingNotePlaintext::decrypt() now has to be aware of consensus params and blockheight. Its callers in wallet, rpcwallet, and tests are updated accordingly.
TransactionBuilder is also modified to reject invalid leadBytes.
Co-authored by Daira Hopwood (daira@jacaranda.org)
Add Foundation's and gtank's DNS seeders
This adds our new DNS seeders to the list. They're running [CoreDNS](https://coredns.io) with a [Zcash crawler plugin](https://github.com/ZcashFoundation/dnsseeder), the result of a Zcash Foundation in-house development effort to replace zcash-seeder with something memory safe and easier to maintain.
These are validly operated seeders per the existing policy (https://zcash.readthedocs.io/en/latest/rtd_pages/dnsseed_policy.html):
> A DNS seed operating organization or person is expected to follow good host security practices, maintain control of applicable infrastructure, and not sell or transfer control of the DNS seed. Any hosting services contracted by the operator are equally expected to uphold these expectations.
In both cases the code is running on well-operated public cloud infrastructure in either a container or the most sandboxing appropriate to the environment. The DNS records pointing to the seeders are controlled by reputable third-party DNS providers under accounts with 2FA enabled.
> The DNS seed results must consist exclusively of fairly selected and functioning Zcash nodes from the public network to the best of the operator’s understanding and capability.
The crawler attempts to connect to all discoverable Zcash peers and ensures their continued uptime on a regular basis. The results are always a uniformly randomized subset of all known live peers.
> For the avoidance of doubt, the results may be randomized but must not single out any group of hosts to receive different results unless due to an urgent technical necessity and disclosed.
See above. However, we reserve the right to begin offering [NU-targeted results](https://github.com/ZcashFoundation/dnsseeder/issues/3) based on opt-in client queries.
> The results may not be served with a DNS TTL of less than one minute.
Mainnet results are served with a TTL of 600 seconds, and Testnet results with a TTL of 300 seconds to account for greater flux on that network.
> Any logging of DNS queries should be only that which is necessary for the operation of the service or urgent health of the Zcash network and must not be retained longer than necessary nor disclosed to any third party.
There is no logging of DNS queries in either production configuration, which can be somewhat confirmed by examining the Corefile(s) [[1]](https://github.com/ZcashFoundation/coredns-zcash/blob/master/coredns/Corefile)[[2]](https://github.com/ZcashFoundation/coredns-zcash/blob/master/scripts/gcp-start.sh#L9-L27) we use.
> Information gathered as a result of the operators node-spidering (not from DNS queries) may be freely published or retained, but only if this data was not made more complete by biasing node connectivity (a violation of expectation (1)).
The seeder currently has no persistence outside of its static config file, so this data is neither retained nor shared by the operators.
> Operators are encouraged, but not required, to publicly document the details of their operating practices.
Our deployments are described in detail by the [coredns-zcash](https://github.com/ZcashFoundation/coredns-zcash) repo. Reader, you could run one too!
> A reachable email contact address must be published for inquiries related to the DNS seed operation.
For general questions related to either seeder, contact george@zfnd.org or mention @gtank in the Foundation's Discord. For bug reports, open an issue on the [dnsseeder](https://github.com/ZcashFoundation/dnsseeder) repo.
Rather than flushing the witness cache from FlushStateToDisk(), called
by ActivateBestChain() called by ProcessNewBlock(), do so from
ThreadNotifyWallets() after the wallet has updated the in-memory witness
data according to the new block, so it's always consistent on disk.
metrics: Add a progress bar when in Initial Block Download mode
The progress bar shows both headers (in green) and blocks (in white / inverse of background colour). It is only printed for TTY output.
Additionally, the "not mining" message is no longer shown on mainnet, as the built-in CPU miner is not effective at the current network difficulty.
This patch adds an option to configure the name and/or directory of the
debug log.
The user can specify either a relative path, in which case the path
is relative to the data directory. They can also specify an absolute
path to put the log anywhere else in the file system.
Report headers download
With current compile-time defaults, a Zcash node prefetches up to 160 block headers per request without a limit on how far it can prefetch, but only up to 16 full blocks at a time. For this and other reasons, it can get very far ahead in headers prefetch (and PoW verification on those, so it's quite some processing too) over full blocks fetch (such as 10x ahead) during initial blocks download. Let's report to the user on how many headers the node has fetched, and let's also use this information as additional input on estimating the total number of blocks to fetch: it can't be less than the number of headers already fetched.
While at it, also fix typos in related code.
Use the cached consensusBranchId in DisconnectBlock
If a node is started with a set of network upgrades that don't match the
serialized chain (such as when we implement NU rollbacks on testnet),
RewindBlockIndex will disconnect each block in the chain until it
reaches the most recent block that agrees with the node's set of network
upgrades. However, the blocks themselves should be disconnected using
the consensus branch ID that they were connected with, which is
persisted alongside the chain and reconstructed in LoadBlockIndex.
If a node is started with a set of network upgrades that don't match the
serialized chain (such as when we implement NU rollbacks on testnet),
RewindBlockIndex will disconnect each block in the chain until it
reaches the most recent block that agrees with the node's set of network
upgrades. However, the blocks themselves should be disconnected using
the consensus branch ID that they were connected with, which is
persisted alongside the chain and reconstructed in LoadBlockIndex.
librustzcash: make the header C compatible
The `librustzcash.h` file is compatible with both languages. However, only C++ is supported at the moment. By relying on the preprocessor to include or not the `extern "C"` piece, the interface becomes compatible with both.
Enable Heartwood activation on mainnet
This sets the Heartwood activation height to `903000`, which follows the deprecation height of `v2.1.2-3` (which is set to deprecate on block `901475`, roughly 31 hours earlier, sometime mid-July).
Fix a null pointer dereference that occurs when formatting an error message
This fixes a bug in the error message printout for the case when we have sufficient chain work, but an expected network upgrade has not activated.
These are reserved for all uses (including member function names) in C++,
and their use is technically undefined behaviour:
https://stackoverflow.com/a/228797/393146
Co-authored-by: Daira Hopwood <daira@jacaranda.org>
When calling GetHistoryRoot, use prevConsensusBranchId instead of consensusBranchId for compatibility with NU4 and future upgrades.
Co-authored by Jack Grigg (jack@electriccoin.co)
#4499 was an insufficient fix, because it did not consider the case where
a post-Heartwood node wrote a block index object for a header from a
non-upgraded peer. In that case the version in the block index entry is
`>= CHAIN_HISTORY_ROOT_VERSION`, and so the fix in #4499 has no effect.
In addition, we should skip the consistency check when the index object
validity is not BLOCK_VALID_CONSENSUS.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Reproduce and fix off-by-one error in reorg logic (#4119)
This attempts to reproduce #4119 using a simple chain split.It currently seems to trigger a different issue, an assertion failure in `CheckBlockIndex` when restarting
If we are doing an expected rollback that changes the consensus
branch ID for some upgrade (or introduces one that wasn't present
at the equivalent height) this will occur because
`SelectHistoryCache` selects the tree for the new consensus
branch ID, not the one that existed on the chain being rolled
back.
In the vicinity of a network upgrade, a zcashd node may receive headers
for a non-upgrading chain from its non-upgraded peers (e.g. if the block
at the NU activation height is found more quickly by the non-upgrading
chain). In this situation, the node will end up with two headers at the
NU activation height (and possibly for subsequent block heights).
In the case of Heartwood, the block headers from the non-upgrading chain
do not satisfy the Heartwood header consistency check in
CBlockTreeDB::LoadBlockIndexGuts. In this commit, we restrict the
Heartwood consistency checks to block index objects that were created by
clients that are CHAIN_HISTORY_ROOT_VERSION or better.
Fix advertised version
Closes https://github.com/zcash/zcash/issues/4375 by adding the `-` character to the list of safe ones.
Now i can see stuff like the following in the logs:
```
...
2020-04-13 14:19:37 receive version message: /MagicBean:2.1.1-1/: version 170009, blocks=795400, us=[2800:a4:316b:8e00:ceb:c7b4:3481:197f]:59754, peer=2
...
```
API call `getpeerinfo` will also gets fixed.
memory_cleanse backports
Cherry-picked from the following upstream PRs:
- bitcoin/bitcoin#10308
- bitcoin/bitcoin#11196
- bitcoin/bitcoin#11558
- Only the changes that did not conflict.
- bitcoin/bitcoin#16158
Part of #145.
So far, the documentation of memory_cleanse() is a verbatim copy of
the commit message in BoringSSL, where this code was originally
written. However, our code evolved since then, and the commit message
is not particularly helpful in the code but is rather of historical
interested in BoringSSL only.
This commit improves improves the comments around memory_cleanse()
and gives a better rationale for the method that we use. This commit
touches only comments.
Commit fbf327b13868861c2877c5754caf5a9816f2603c ("Minimal code
changes to allow msvc compilation.") was indeed minimal in terms
of lines touched. But as a result of that minimalism it changed the
logic in memory_cleanse() to first call std::memset() and then
additionally the MSVC-specific SecureZeroMemory() function, and it
also moved a comment to the wrong location.
This commit removes the superfluous call to std::memset() on MSVC
and ensures that the comment is in the right position again.
The implementation we currently use from OpenSSL prevents the compiler from optimizing away clensing operations on blocks of memory that are about to be released, but this protection is not extended to link-time optimization. This commit copies the solution cooked up by Google compiler engineers which uses inline assembly directives to instruct the compiler not to optimize out the call under any circumstances. As the code is in-lined, this has the added advantage of removing one more OpenSSL dependency.
Regarding license compatibility, Google's contributions to BoringSSL library, including this code, is made available under the ISC license, which is MIT compatible.
BoringSSL git commit: ad1907fe73334d6c696c8539646c21b11178f20f
Add confirmations, blockheight, blockindex and blocktime to z_listreceivedbyaddress
Fixes https://github.com/zcash/zcash/issues/3724
1- There was a PR to add confirmations to this call at https://github.com/zcash/zcash/pull/3836
I ported the commit from there and fixed test case by incrementing the confirmations as suggested at: https://github.com/zcash/zcash/pull/3836#issuecomment-499927807
2- Then added `blockheight`, `blockindex` and `blocktime`. To avoid some duplicated code (Sprout/Sapling) created a structure `trxblock`.
3- Original issue requests only time and blockindex however i think height is also important; if `blockindex` is the position of the transaction in the block then you are going to need also `height` to find it.
- hashFinalSaplingRoot is now always set to hashLightClientRoot before
Heartwood activation (regardless of whether or not that is the correct
Sapling root), and set to the Sapling root after Heartwood activation
only for blocks that have been passed to ConnectBlock() at least once.
- hashChainHistoryRoot is now always set "correctly" (either null, or
identical to hashLightClientRoot).
We rely on the fact that block headers are downloaded in order, and we
therefore always know the height of a block header, in order to check
whether Heartwood is active for a particular header.
Compute more structures in CTxMemPool::DynamicMemoryUsage()
Closes https://github.com/zcash/zcash/issues/4224
Added more structures to the computation of the mempool memory size as indicated in the issue. RPC call `getmempoolinfo` is the only place where this is used to get the `usage` result.
Lock with cs_main inside gtests that request chain state
Fix https://github.com/zcash/zcash/issues/4389
- Used the lock from boost tests where `chainActive.Height()` is being called: https://github.com/zcash/zcash/blob/master/src/wallet/test/rpc_wallet_tests.cpp#L1323
- I found no other place in the gtests where `chainActive` is used apart from just the same tests `chainActive.Height()` is called. It seems chain state is only used when we fake mine transactions and these are all inside `test_wallet.cpp`.
I might be missing some other patterns to look at, please let me know if so.
The C++ and Rust Equihash validators are intended to have an identical
set of valid Equihash solutions, so this should merely be an
implementation detail. However, deploying the Rust validator at the same
time as a network upgrade reduces the risk of an unintentional consensus
divergence due to undocumented behaviour in either implementation.
Once Heartwood has activated on mainnet, we can verify that all
pre-Heartwood blocks satisfy the Rust validator, and then remove the C++
validator and make Equihash-checking non-contextual again.
This requires moving CheckEquihashSolution() to
ContextualCheckBlockHeader() for all but the genesis block, which has no
effect on consensus; it just means that an invalid Equihash solution is
rejected slightly later in the block validation process.
Quoting the documentation for `std::vector::operator[]`:
Portable programs should never call this function with an argument
n that is out of range, since this causes undefined behavior.
This test was doing just that: performing checks on a non-existent
second Sapling witness (duplicating the Sprout logic that checked two
notes, one of which was in the wallet). The test was instead reading
arbitrary memory after the witness that did exist; in most cases, this
memory was interpreted as a `boost::none` as expected, but in some cases
the memory was interpreted as a "real" witness.
Closeszcash/zcash#4445.
Co-authored-by: Ying Tong <yingtong@ethereum.org>
Add -lightwalletd experimental option
Similar to `-insightexplorer` but loading less indexes.
After testing and code review this should be able to close https://github.com/zcash/zcash/issues/4326
multiple debug categories documentation
The command line help is not clear on how to execute multiple but specific debug categories. Failed with stuff like `-zcashd -debug=category1, category2`, etc to find how to do it after some time.
The proposed line addition should help with that.
Check failing transparent and JoinSplit signatures against the previous network upgrade
This change improves usability across network upgrades, by informing
users when their new transactions are being created with the consensus
branch ID from the previous epoch.
We only check failing signatures against the previous epoch to minimise
the extra computational load on nodes.
A future refactor is needed to similarly check Sapling signatures.
Part of #4260.
The previous version of full_test_suite.py directly called the test
binary, which was being compiled at the same time as the static library.
However, by passing the --tests argument to cargo, rustc was ignoring
several important release-profile configurations, and was also
attempting to link the test binary, which was breaking cross-compilation
builds.
This commit alters src/Makefile.am to only build the static library, and
leaves test compilation to the test runner itself. This ensures that the
tests are only compiled for native builds, when the tests will be run on
the same platform.
std::vector<T> is guaranteed to store T contiguously. However, there is
no guarantee that sizeof(std::array<unsigned char, N>) == N, which
prevents us from interpreting std::vector<std::array<unsigned char, N>>
as &[[u8; N]] on the Rust side of the FFI.
Instead, we define HistoryEntry as a struct wrapping a C array, which
(as checked by static_assert) contains no padding.
The "finalsaplingroothash" field of the getblocktemplate output is no
longer guaranteed to match the actual Sapling commitment tree root, and
has been deprecated. Users should migrate to "lightclientroothash".
CBlockHeader.hashFinalSaplingRoot has been renamed to hashLightClient.
CBlockIndex now stores:
- hashLightClient as from the block header
- hashFinalSaplingRoot, which is accurate for all blocks prior to
Heartwood activation, and all blocks from Heartwood activation onward
that are connected at some point to the main chain in ConnectBlock().
- hashChainHistoryRoot, which is null prior to Heartwood activation, and
set per ZIP 221 from Heartwood activation.
The new block index fields are only written to disk for client version
2.1.2 and above, which will be the first Heartwood-aware clients (even
if Heartwood doesn't have an activation height).
Explanation: if a signal interrupts certain syscalls such as `open`, `read`, or `write`,
then the library function will by default fail with `errno` `EINTR`. But we almost never
check for `EINTR`, so this is likely to cause spurious errors. We want to restart the syscall
instead, which is what `SA_RESTART` is intended to do. Since our signal handlers (defined
in init.cpp) only set a flag, restarting the syscall is safe and is always the Right Thing.
See <https://www.gnu.org/software/libc/manual/html_node/Flags-for-Sigaction.html> and
<https://www.gnu.org/software/libc/manual/html_node/Interrupted-Primitives.html> for
further information.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Add support for Sapling full viewing keys
This PR adds Sapling support to `z_exportviewingkey` and `z_importviewingkey`, and stores imported Sapling viewing keys in the wallet.
Closes#3060.
Additional librustzcash integration
This adds librustzcash tests to the full test suite, and brings in the release profile configurations that are currently present in the librustzcash workspace on the other repository. It's very important that we build librustzcash with panic=abort because otherwise the unwinding panics across FFI boundaries could cause undefined behavior.
Fix race conditions during init
Fix race conditions due to accessing `chainActive.Tip()` during init, and other minor cleanups.
Includes backport of https://github.com/bitcoin/bitcoin/pull/8063 .
Fix typos/minor error in comments, and wrap some lines
This minimizes the diff in the example implementation of funding streams in ZIP 207.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Shows reindex progress in metrics screen
Resolves issue #3813.
The "Downloading blocks" message is changed to "Reindexing blocks" during reindex, after reindex is completed the text is reverted back to the first variant.
Reindex progress is shown as a sum of processed file size (we can't use reindexed block number as a progress because we can't predict how many blocks to process at all, we don't know the size of the block before we process it), the result looks like
```
Reindexing blocks | 22.64 MiB / 336.00 MiB (6%, 13583 blocks)
```
This change improves usability across network upgrades, by informing
users when their new transactions are being created with the consensus
branch ID from the previous epoch.
We only check failing signatures against the previous epoch to minimise
the extra computational load on nodes.
Refactor experimental feature handling
Adds new rpc call `getexperimentalfeatures` and also adds experimental features to `getblockchaininfo` output.
Closes#2671.
Bring the librustzcash crate into this repository
Rust dependencies are now canonically pinned within this repository by
`Cargo.lock`. We continue to use the depends system for vendoring the
dependencies, to ensure our Gitian builds continue to function (which have
no network access at build time, and fetch dependencies separately).
The `--enable-online-rust` configure flag replicates the behaviour of the
`LIBRUSTZCASH_OVERRIDE` environment variable (enabling the build system to
use https://crates.io instead of vendored dependencies).
This pulls in the exact version of `librustzcash` that we currently depend on
(corresponding to the `0.1.0` tag in https://github.com/zcash/librustzcash).
The changes to the crate since then will be pulled in as a separate PR.
Part of zcash/librustzcash#155.
Part of #4230.
Upgrade libsodium to 1.0.18
Includes patches that maintain consensus compatibility with libsodium 1.0.15 for Ed25519 pubkey and signature validation.
Replaces #4239. Closes#2872.
Remove time adjustment; instead warn if peer clocks are too different
The policy is: warn if we have seen at least 8 (TIMEDATA_WARNING_SAMPLES) peer times, in the version messages of the first 20 (TIMEDATA_MAX_SAMPLES) unique (by IP address) peers that connect, that are more than 10 minutes (TIMEDATA_WARNING_THRESHOLD seconds) but less than 10 days (TIMEDATA_IGNORE_THRESHOLD seconds) away from local time.
fixes#4338