multiple debug categories documentation
The command line help is not clear on how to execute multiple but specific debug categories. Failed with stuff like `-zcashd -debug=category1, category2`, etc to find how to do it after some time.
The proposed line addition should help with that.
Check failing transparent and JoinSplit signatures against the previous network upgrade
This change improves usability across network upgrades, by informing
users when their new transactions are being created with the consensus
branch ID from the previous epoch.
We only check failing signatures against the previous epoch to minimise
the extra computational load on nodes.
A future refactor is needed to similarly check Sapling signatures.
Part of #4260.
The previous version of full_test_suite.py directly called the test
binary, which was being compiled at the same time as the static library.
However, by passing the --tests argument to cargo, rustc was ignoring
several important release-profile configurations, and was also
attempting to link the test binary, which was breaking cross-compilation
builds.
This commit alters src/Makefile.am to only build the static library, and
leaves test compilation to the test runner itself. This ensures that the
tests are only compiled for native builds, when the tests will be run on
the same platform.
std::vector<T> is guaranteed to store T contiguously. However, there is
no guarantee that sizeof(std::array<unsigned char, N>) == N, which
prevents us from interpreting std::vector<std::array<unsigned char, N>>
as &[[u8; N]] on the Rust side of the FFI.
Instead, we define HistoryEntry as a struct wrapping a C array, which
(as checked by static_assert) contains no padding.
The "finalsaplingroothash" field of the getblocktemplate output is no
longer guaranteed to match the actual Sapling commitment tree root, and
has been deprecated. Users should migrate to "lightclientroothash".
CBlockHeader.hashFinalSaplingRoot has been renamed to hashLightClient.
CBlockIndex now stores:
- hashLightClient as from the block header
- hashFinalSaplingRoot, which is accurate for all blocks prior to
Heartwood activation, and all blocks from Heartwood activation onward
that are connected at some point to the main chain in ConnectBlock().
- hashChainHistoryRoot, which is null prior to Heartwood activation, and
set per ZIP 221 from Heartwood activation.
The new block index fields are only written to disk for client version
2.1.2 and above, which will be the first Heartwood-aware clients (even
if Heartwood doesn't have an activation height).
Explanation: if a signal interrupts certain syscalls such as `open`, `read`, or `write`,
then the library function will by default fail with `errno` `EINTR`. But we almost never
check for `EINTR`, so this is likely to cause spurious errors. We want to restart the syscall
instead, which is what `SA_RESTART` is intended to do. Since our signal handlers (defined
in init.cpp) only set a flag, restarting the syscall is safe and is always the Right Thing.
See <https://www.gnu.org/software/libc/manual/html_node/Flags-for-Sigaction.html> and
<https://www.gnu.org/software/libc/manual/html_node/Interrupted-Primitives.html> for
further information.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Add support for Sapling full viewing keys
This PR adds Sapling support to `z_exportviewingkey` and `z_importviewingkey`, and stores imported Sapling viewing keys in the wallet.
Closes#3060.
Additional librustzcash integration
This adds librustzcash tests to the full test suite, and brings in the release profile configurations that are currently present in the librustzcash workspace on the other repository. It's very important that we build librustzcash with panic=abort because otherwise the unwinding panics across FFI boundaries could cause undefined behavior.
Fix race conditions during init
Fix race conditions due to accessing `chainActive.Tip()` during init, and other minor cleanups.
Includes backport of https://github.com/bitcoin/bitcoin/pull/8063 .
Fix typos/minor error in comments, and wrap some lines
This minimizes the diff in the example implementation of funding streams in ZIP 207.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Shows reindex progress in metrics screen
Resolves issue #3813.
The "Downloading blocks" message is changed to "Reindexing blocks" during reindex, after reindex is completed the text is reverted back to the first variant.
Reindex progress is shown as a sum of processed file size (we can't use reindexed block number as a progress because we can't predict how many blocks to process at all, we don't know the size of the block before we process it), the result looks like
```
Reindexing blocks | 22.64 MiB / 336.00 MiB (6%, 13583 blocks)
```
This change improves usability across network upgrades, by informing
users when their new transactions are being created with the consensus
branch ID from the previous epoch.
We only check failing signatures against the previous epoch to minimise
the extra computational load on nodes.
Refactor experimental feature handling
Adds new rpc call `getexperimentalfeatures` and also adds experimental features to `getblockchaininfo` output.
Closes#2671.
Bring the librustzcash crate into this repository
Rust dependencies are now canonically pinned within this repository by
`Cargo.lock`. We continue to use the depends system for vendoring the
dependencies, to ensure our Gitian builds continue to function (which have
no network access at build time, and fetch dependencies separately).
The `--enable-online-rust` configure flag replicates the behaviour of the
`LIBRUSTZCASH_OVERRIDE` environment variable (enabling the build system to
use https://crates.io instead of vendored dependencies).
This pulls in the exact version of `librustzcash` that we currently depend on
(corresponding to the `0.1.0` tag in https://github.com/zcash/librustzcash).
The changes to the crate since then will be pulled in as a separate PR.
Part of zcash/librustzcash#155.
Part of #4230.
Upgrade libsodium to 1.0.18
Includes patches that maintain consensus compatibility with libsodium 1.0.15 for Ed25519 pubkey and signature validation.
Replaces #4239. Closes#2872.
Remove time adjustment; instead warn if peer clocks are too different
The policy is: warn if we have seen at least 8 (TIMEDATA_WARNING_SAMPLES) peer times, in the version messages of the first 20 (TIMEDATA_MAX_SAMPLES) unique (by IP address) peers that connect, that are more than 10 minutes (TIMEDATA_WARNING_THRESHOLD seconds) but less than 10 days (TIMEDATA_IGNORE_THRESHOLD seconds) away from local time.
fixes#4338
This enables IDE integration to work (which requires the Cargo.toml to
be in the repo root).
"make clean" no longer runs "cargo clean", because IDE integrations hold
locks on files within the Rust build directory, and an error inside
"cargo clean" error would prevent "make clean" from completing (and
removing other files).
The --enable-online-rust configure flag replicates the behaviour of the
LIBRUSTZCASH_OVERRIDE environment variable (enabling the build system to
use crates.io instead of vendored dependencies).
The Cargo.lock is updated to account for the Zcash Rust crates no longer
being in the same workspace. Dependencies that were not transitive
dependencies of librustzcash are also removed, but no versions change.
Allow negative heights in RPC calls
For issue https://github.com/zcash/zcash/issues/2197
Currently adding the feature to `getblock` and `getblockhash`. There is another candidate: `getblocksubsidy` however i want to have some review about these 2 first before repeating what could be a bad approach.
Initialize ThreadNotifyWallets before additional blocks are imported
The intention of this PR is to address #4301. It seems `ThreadNotifyWallets` assumes that `chainActive.Tip()` on initialization reflects the wallet's view of the active tip, but prior to `ThreadNotifyWallets`'s initialization the active chain ends up being modified by blocks that are imported from disk, so our local `pindexLastTip` ends up out of sync with the wallet's actual "last" tip.
This PR moves the initialization of `ThreadNotifyWallets` to earlier in the process while passing in the current active tip. This addresses the problem for non-fresh nodes, and then for fresh nodes we have the notification thread wait until the genesis block has been loaded.
Change cm to cmu in sapling
Part of https://github.com/zcash/zcash/issues/3446
For each of the 2 commits, suggested change was made, then fixed build errors until compile. Finally ran bitcoin and gtests, both of them passing.
Upstream PRs relating to strMiscWarning
This pulls in upstream PRs bitcoin/bitcoin#7114 and bitcoin/bitcoin#9236 (non-QT parts).
* Fixesbitcoin/bitcoin#6809 - run-of-the-mill exceptions should not get into `strMiscWarning` (which is reported by `getinfo`).
* Eliminate data races for `strMiscWarning` and `fLargeWork*Found`. This moves all access to these data structures through accessor functions and protects them with a lock.
This moves all access to these datastructures through accessor functions
and protects them with a lock.
Relative to the upstream commit, we also add GetMiscWarning() to make this accessible to tests.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
This is a first step in avoiding racy accesses to strMiscWarning.
By itself this commit causes a link error because global variables relating to alerts
are only accessible in the zcashd server, but util is used in other binaries.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
All of these maps are created from scratch on wallet load, so we can
alter their contents without compatibility concerns. With this change,
we can use the existing maps to implement viewing key support. The
downside is that the wallet will take more space in memory, but that can
easily be improved in future by storing ExtFVK fingerprints in some of
the maps (which would improve memory usage even compared to the original
layout).
z_viewtransaction
This RPC method returns all decryptable information for any transaction in the wallet.
Several values are conditionally included in the output for convenience:
- `recovered`: True if an output is not for a Sapling address in the wallet.
- `memoStr`: The text form of an output's memo, if it is valid UTF-8.
- Values are provided both in decimal currency units, and integer zatoshis.
I'm not adding test vectors for non-canonical pubkeys, as that would
require grinding to find a private key corresponding to one of the 19
pubkeys that can be non-canonical.
Reduce duplication in key_io decode
Attempt to fix https://github.com/zcash/zcash/issues/3344 by adding a new `DecodeAny` template function with arguments that will handle all the cases. Then on each decoding this new function is called with the appropriate arguments resulting in reducing code duplication.
Some of the complexity(`boost::optional`) will be removed if `ViewingKey` is implemented for sapling.
Worst-case block verification benchmarks
The micro-benchmark framework from #3858 is used to measure the time to verify various transaction components. These are leveraged by a script that simulates the verification cost of different compositions of full blocks.
Add z_getnewaddress test
Supercedes #3749. Closes#3687.
It takes the commit from @dagurval where a new test case
`rpc_wallet_z_getnewaddress` is added and it adds a check for too many
arguments.
It uses `CheckRPCThrows()` to test the error message of invalid argument.
However, for too many arguments the full help message is displayed, so I
am just testing there is an runtime exception.
Note that the MTP of a block is the median timestamp of the preceding 11 blocks, i.e. it is
typically (with no or only moderate timestamp manipulation) expected to be 6 block intervals
behind that block's timestamp, which *on average* is 450 seconds behind (after Blossom activation).
So the effective limit on future dating of timestamps is ~82.5 minutes. This makes it
exceptionally unlikely --even taking into account feasible timestamp manipulation of this
and previous blocks-- that the chain will stall because no block is found before the limit.
(This may rely on assumptions that do not hold for testnet.)
If an adversary were to have a sufficient fraction of mining power to engineer this situation
then there would be something seriously wrong, and arguably the chain should stall in that
case, pending manual intervention.
Co-authored-by: Jack Grigg <jack@z.cash>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
The use of virtual on these lines is obsolete.
I ran:
"rm -f ./src/zcash-gtest && rm -f ./src/gtest/*.o && make && ./src/zcash-gtest"
before an after making the change. In both cases:
206 test ran
from
32 cases
and
1 test was DISABLED
This allows for fPIE to be used selectively.
Zcash: Second half of upstream commit 17c4d9d1647bbac4b0557136b1c3d98c951feb79
First half was pulled in as c459de2f03
lastCycles was introduced in 35328187463a7078b4206e394c21d5515929c7de which was merged into master yesterday.
Also initialize beginCycles to zero for consistency and completeness.
std::chrono removes portability issues.
Rather than storing doubles, store the untouched time_points. Then
convert to nanoseconds for display. This allows for maximum precision, while
keeping results comparable between differing hardware/operating systems.
Also, display full nanosecond counts rather than sub-second floats.
We were saving a div by caching the inverse as a float, but this
ended up requiring a int -> float -> int conversion, which takes
almost as much time as the difference between float mul and div.
There are lots of other more pressing issues with the bench
framework which probably require simply removing the adaptive
iteration count stuff anyway.
Avoid static analyzer warnings regarding "Function call argument
is a pointer to uninitialized value" in cases where we are
intentionally using such arguments.
This is achieved by using ...
`f(b.begin(), b.end())` (`std::array<char, N>`)
... instead of ...
`f(b, b + N)` (`char b[N]`)
Rationale:
* Reduce false positives by guiding static analyzers regarding our
intentions.
Before this commit:
```
$ clang-tidy-3.5 -checks=* src/bench/base58.cpp
bench/base58.cpp:23:9: warning: Function call argument is a pointer to uninitialized value [clang-analyzer-core.CallAndMessage]
EncodeBase58(b, b + 32);
^
$ clang-tidy-3.5 -checks=* src/bench/verify_script.cpp
bench/verify_script.cpp:59:5: warning: Function call argument is a pointer to uninitialized value [clang-analyzer-core.CallAndMessage]
key.Set(vchKey, vchKey + 32, false);
^
$
```
After this commit:
```
$ clang-tidy-3.5 -checks=* src/bench/base58.cpp
$ clang-tidy-3.5 -checks=* src/bench/verify_script.cpp
$
```
Zcash: Only applied changes to src/bench/base58.cpp
The initialization order of global data structures in different
implementation units is undefined. Making use of this is essentially
gambling on what the linker does, the so-called [Static initialization
order fiasco](https://isocpp.org/wiki/faq/ctors#static-init-order).
In this case it apparently worked on Linux but failed on OpenBSD and
FreeBSD.
To create it on first use, make the registration structure local to
a function.
Fixes#8910.
Make sure that the count is a zero modulo the new mask before
scaling, otherwise the next time until a measure triggers
will take only 1/2 as long as accounted for. This caused
the 'min time' to be potentially off by as much as 100%.
There are only a few uses of `insecure_random` outside the tests.
This PR replaces uses of insecure_random (and its accompanying global
state) in the core code with an FastRandomContext that is automatically
seeded on creation.
This is meant to be used for inner loops. The FastRandomContext
can be in the outer scope, or the class itself, then rand32() is used
inside the loop. Useful e.g. for pushing addresses in CNode or the fee
rounding, or randomization for coin selection.
As a context is created per purpose, thus it gets rid of
cross-thread unprotected shared usage of a single set of globals, this
should also get rid of the potential race conditions.
- I'd say TxMempool::check is not called enough to warrant using a special
fast random context, this is switched to GetRand() (open for
discussion...)
- The use of `insecure_rand` in ConnectThroughProxy has been replaced by
an atomic integer counter. The only goal here is to have a different
credentials pair for each connection to go on a different Tor circuit,
it does not need to be random nor unpredictable.
- To avoid having a FastRandomContext on every CNode, the context is
passed into PushAddress as appropriate.
There remains an insecure_random for test usage in `test_random.h`.
Zcash: Resolved conflicts with the following files
src/addrman.cpp
src/main.cpp
src/net.cpp
src/net.h
src/policy/fees.cpp
src/policy/fees.h
src/random.cpp
src/test/merkle_tests.cpp
src/test/net_tests.cpp
src/test/prevector_tests.cpp
src/test/sighash_tests.cpp
src/test/skiplist_tests.cpp
src/test/test_bitcoin.cpp
src/test/versionbits_tests.cpp
src/wallet/test/crypto_tests.cpp
Previously the benchmark code used an integer division (%) with
a non-constant in the inner-loop. This is quite slow on many
processors, especially ones like ARM that lack a hardware divide.
Even on fairly recent x86_64 like haswell an integer division can
take something like 100 cycles-- making it comparable to the
runtime of siphash.
This change avoids the division by using bitmasking instead. This
was especially easy since the count was only increased by doubling.
This change also restarts the timing when the execution time was
very low this avoids mintimes of zero in cases where one execution
ends up below the timer resolution. It also reduces the impact of
the overhead on the final result.
The formatting of the prints is changed to not use scientific
notation make it more machine readable (in particular, gnuplot
croaks on the non-fixedpoint, and it doesn't sort correctly).
This also hoists out all the floating point divisions out of the
semi-hot path because it was easy to do so.
It might be prudent to break out the critical test into a macro
just to guarantee that it gets inlined. It might also make sense
to just save out the intermediate counts and times and get the
floating point completely out of the timing loop (because e.g.
on hardware without a fast hardware FPU like some ARM it will
still be slow enough to distort the results). I haven't done
either of these in this commit.
- ensure header namespaces and end comments are correct
- add missing header end comments
- ensure minimal formatting (add newlines etc.)
Zcash: left out change to src/policy/policy.h which we don't yet have.
Avoid calling gettimeofday every time through the benchmarking loop, by keeping
track of how long each loop takes and doubling the number of iterations done
between time checks when they take less than 1/16'th of the total elapsed time.
Benchmarking framework, loosely based on google's micro-benchmarking
library (https://github.com/google/benchmark)
Wny not use the Google Benchmark framework? Because adding Even More Dependencies
isn't worth it. If we get a dozen or three benchmarks and need nanosecond-accurate
timings of threaded code then switching to the full-blown Google Benchmark library
should be considered.
The benchmark framework is hard-coded to run each benchmark for one wall-clock second,
and then spits out .csv-format timing information to stdout. It is left as an
exercise for later (or maybe never) to add command-line arguments to specify which
benchmark(s) to run, how long to run them for, how to format results, etc etc etc.
Again, see the Google Benchmark framework for where that might end up.
See src/bench/MilliSleep.cpp for a sanity-test benchmark that just benchmarks
'sleep 100 milliseconds.'
To compile and run benchmarks:
cd src; make bench
Sample output:
Benchmark,count,min,max,average
Sleep100ms,10,0.101854,0.105059,0.103881
Such outputs can still be watched, and signed for, but they aren't treated as valid payments.
That means they won't cause transactions to appear in listtransactions, their outputs to be
shown under listunspent, or affect balances.
Inside IsMine we care about the distinction between scriptPubKey execution
and P2SH redeemScript execution. The consensus code does not care about this
distinction, and thus SigVersion does not have a field for P2SH. As the IsMine
code will care, it uses a separate enum with more fields.
Only IsMine's internal code needs this, as part of a recursion into P2SH and P2WSH
scripts. The exposed functions always operate on actual scriptPubKeys and not on
redeemScripts or witness scripts.
Zcash: Rewrote this commit so that it instead adds an internal
SigVersion argument (which we didn't have because it was added to
support SegWit logic).
Co-authored-by: Jack Grigg <jack@z.cash>
Before this, if someone imported a scriptPubKey directly (in hex form) using
importaddress, outputs sending to it would be treated as change, as the
corresponding CTxDestination was not added to the address book.
Fix this by trying to detect scriptPubKeys that are in fact convertible to a
CTxDestination and add them anyway. Add a warning to the RPC help to warn
against importing raw non-standard scripts.
Wallet interface refactor
This refactors the logic introduced in #4144 to improve the separation between the node and wallet. The notifier thread now lives next in `src/validationinterface.cpp` directly next to the existing `CMainSignals` node-wallet interface.
Part of #3877.
ThreadNotifyWallets now collects all notification-related state at once,
and then executes wallet logic based on the collected state after
dropping any locks it is holding.
Jack Grigg
Homu
Alfredo Garcia
Dimitris Apostolou
NikVolf
Daira Hopwood
Sean Bowe
Patrick Strateman
gladcow
MeshCollider
Larry Ruane
Taylor Hornby
Gregory Maxwell
Wladimir J. van der Laan
mdr0id
zancas
practicalswift
Cory Fields
Matt Corallo
Pavel Janík
Pieter Wuille
Jeremy Rubin
Karl-Johan Alm
Yuri Zhykin
Philip Kaufmann
Gavin Andresen
Dagur Valberg Johannsson
sandakersmann
Jim Posen
Jonas Schnelli
MarcoFalke
Denis Lukianov
Luke Dashjr