Flush witness data to disk only when it's consistent
Closes#4301. Running this PR's code will not repair a data directory that has been affected by this problem; that requires starting zcashd with the `-rescan` or `-reindex` options.
This makes the test more deterministic. When the nodes are split, and
partition B (node 1) mines a joinsplit transaction, that block only
sometimes appears on the blockchain after the nodes are all reconnected.
ZIP212 implementation
Closes#4557.
(description by @ebfull, taken from #4575)
* The `SaplingNote` structure has a new enum called `zip212Enabled`. This
member is private and reflects whether the note was or is being created
using the derivation method of ZIP 212 (i.e., `BeforeZip212` or `AfterZip212`).
* The `SaplingNotePlaintext` structure has a new unsigned char member
`leadbyte`. This member is private and contains the leading byte of the
plaintext (e.g. `0x01`, `0x02`).
* The serialization of `SaplingNotePlaintext` sets `zip212Enabled` to
`BeforeZip212` iff the serialized note plaintext version is not `0x01`.
* The `r`/`rcm` fields have been removed and replaced with a private field
`rseed`. `SaplingNote` and `SaplingNotePlaintext` now have a helper method
`rcm()` which returns the `rcm` either by deriving it with `rseed`
(if `zip212Enabled` is `AfterZip212`) or returning `rseed` by interpreting
`rseed` as `rcm`.
* All the methods of obtaining a `SaplingNote` account for these changes:
- The `SaplingNote` constructor that is used by e.g. the transaction builder,
and internally samples random `rcm`, now takes a `zip212Enabled` argument
to decide whether to sample `rcm` the "old" way or the "new" way.
- The bare constructor for `SaplingNote` is removed.
- The other constructor which takes the raw contents of the note is only used
in tests or in `Note.cpp`, but now also takes a `zip212Enabled` argument.
- The other way of obtaining a note, by calling `SaplingNotePlaintext::note()`,
has been adjusted.
* The `SaplingNotePlaintext` class now has an `generate_or_derive_esk()` method
that either samples a random `esk` or derives it using the local `rseed`
depending on the value of `leadbyte`.
* The encryption routine is modified to consult `generate_or_derive_esk()` and
provide it to the note encryption object.
* The note encryption objects now take an optional `esk` as input and otherwise
sample a random `esk` internally. This API functionality is preserved to allow
for testing.
* The `SaplingNotePlaintext` decryption routines are modified:
- The out and enc decryption routines now check that `epk` is consistent with
the derived `esk`.
- The out decryption routine for plaintexts also checks that `esk` is
consistent with what is derived by the note.
* The miner and transaction builder consult the activation of Canopy when
creating `SaplingNote`s.
* The consensus rules are modified so that shielded outputs (miner rewards)
must have `v2` note plaintexts after Canopy has activated.
This fixes wallet_anchorfork.py CI failure, but a separate PR
will restore flushing witness data on shutdown while also
fixing DecrementNoteWitnesses() to not assert when
nd->witnessHeight < indexHeight, which can happen when the
node reorgs upon restart (which this test causes to happen).
We only relied on success being 0 and our code was otherwise agnostic to the
actual return code in the event of failed signature verification, but this
change keeps the API consistent.
[ZIP 211] Disabling Addition of New Value to the Sprout Value Pool
Disables Sprout outputs after NU4 by checking for nonzero `vpub_old` in transactions after NU4 activation height.
Adds gtests to check expected behaviour before and after NU4 activation height.
edit:
Also modifies `z_` methods in `rpcwallet`, and adds a matching RPC test.
Implements [ZIP 211](https://zips.z.cash/zip-0211), closes#4479
Add joinSplitPubKey and joinSplitSig to RPC
These two properties are required to fully reconstruct a Zcash transaction's binary form from the RPC data.
SaplingNotePlaintext::decrypt() now has to be aware of consensus params and blockheight. Its callers in wallet, rpcwallet, and tests are updated accordingly.
TransactionBuilder is also modified to reject invalid leadBytes.
Co-authored by Daira Hopwood (daira@jacaranda.org)
Add Foundation's and gtank's DNS seeders
This adds our new DNS seeders to the list. They're running [CoreDNS](https://coredns.io) with a [Zcash crawler plugin](https://github.com/ZcashFoundation/dnsseeder), the result of a Zcash Foundation in-house development effort to replace zcash-seeder with something memory safe and easier to maintain.
These are validly operated seeders per the existing policy (https://zcash.readthedocs.io/en/latest/rtd_pages/dnsseed_policy.html):
> A DNS seed operating organization or person is expected to follow good host security practices, maintain control of applicable infrastructure, and not sell or transfer control of the DNS seed. Any hosting services contracted by the operator are equally expected to uphold these expectations.
In both cases the code is running on well-operated public cloud infrastructure in either a container or the most sandboxing appropriate to the environment. The DNS records pointing to the seeders are controlled by reputable third-party DNS providers under accounts with 2FA enabled.
> The DNS seed results must consist exclusively of fairly selected and functioning Zcash nodes from the public network to the best of the operator’s understanding and capability.
The crawler attempts to connect to all discoverable Zcash peers and ensures their continued uptime on a regular basis. The results are always a uniformly randomized subset of all known live peers.
> For the avoidance of doubt, the results may be randomized but must not single out any group of hosts to receive different results unless due to an urgent technical necessity and disclosed.
See above. However, we reserve the right to begin offering [NU-targeted results](https://github.com/ZcashFoundation/dnsseeder/issues/3) based on opt-in client queries.
> The results may not be served with a DNS TTL of less than one minute.
Mainnet results are served with a TTL of 600 seconds, and Testnet results with a TTL of 300 seconds to account for greater flux on that network.
> Any logging of DNS queries should be only that which is necessary for the operation of the service or urgent health of the Zcash network and must not be retained longer than necessary nor disclosed to any third party.
There is no logging of DNS queries in either production configuration, which can be somewhat confirmed by examining the Corefile(s) [[1]](https://github.com/ZcashFoundation/coredns-zcash/blob/master/coredns/Corefile)[[2]](https://github.com/ZcashFoundation/coredns-zcash/blob/master/scripts/gcp-start.sh#L9-L27) we use.
> Information gathered as a result of the operators node-spidering (not from DNS queries) may be freely published or retained, but only if this data was not made more complete by biasing node connectivity (a violation of expectation (1)).
The seeder currently has no persistence outside of its static config file, so this data is neither retained nor shared by the operators.
> Operators are encouraged, but not required, to publicly document the details of their operating practices.
Our deployments are described in detail by the [coredns-zcash](https://github.com/ZcashFoundation/coredns-zcash) repo. Reader, you could run one too!
> A reachable email contact address must be published for inquiries related to the DNS seed operation.
For general questions related to either seeder, contact george@zfnd.org or mention @gtank in the Foundation's Discord. For bug reports, open an issue on the [dnsseeder](https://github.com/ZcashFoundation/dnsseeder) repo.
Rather than flushing the witness cache from FlushStateToDisk(), called
by ActivateBestChain() called by ProcessNewBlock(), do so from
ThreadNotifyWallets() after the wallet has updated the in-memory witness
data according to the new block, so it's always consistent on disk.
Homu
Larry Ruane
therealyingtong
Sean Bowe
Alfredo Garcia
Kris Nuttycombe
therealyingtong
Rod Vagg