Split wallet.py
Split some of the tests from `wallet.py` into its own files to make it easier to debug.
Some of the tests moved where depending(specially in balance amounts) on previous code. As standalone, some of the hardcoded balances needed some changes. No further modifications to tests are done in this PR.
We can split more but i think is a good start.
If this is merged it will close https://github.com/zcash/zcash/issues/4917
-----------------------
Please ensure this checklist is followed for any pull requests for this repo. This checklist must be checked by both the PR creator and by anyone who reviews the PR.
* [ ] Relevant documentation for this PR has to be completed and reviewed by @mdr0id before the PR can be merged
* [ ] A test plan for the PR must be documented in the PR notes and included in the test plan for the next regular release
As a note, all buildbot tests need to be passing and all appropriate code reviews need to be done before this PR can be merged
qa/zcash/updatecheck.py: remove dead code; print instructions to run `cargo outdated` and `cargo update`
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Reduce default fee to 1000 zatoshis
Per ZIP 313. This also ensures that transactions that pay the default fee will always be relayed, and not rate-limited.
This removes the paches iostreams-106.patch and signals2-noise.patch
which have been incorporated into boost 1.75. Also, this further
postpones updates to native_clank, libcxx and native_ccache.
A few miscellaneous improvements to rpc-tests.py command line arguments:
- make all arguments start with double dash for consistency
- improve help text and output
- add nozmq argument to explicitly exclude the ZMQ tests
- change 'parallel' to 'jobs'
- Link pull-tester/rpc-tests.py to the build dir
- Add the build-dir's config to the python path so that tests can find it
- The tests themselves are in srcdir
- Clean up __pycache__ in 'make clean'
Previously these functions would infinitely loop if sync failed;
now they have a default timeout of 60 seconds, after which an
AssertionError is raised.
sync_blocks() has also been improved and now compares the tip
hash of each node, rather than just using block count.
Zcash: Kept block count check for a couple of tests where we use it.
If ZMQ is enabled, check whether it's installed before running ZMQ tests.
If it isn't, disable ZMQ and print a warning.
Also add dependency info to test docs, so users know ZMQ is required
before running tests, and so they know how to install it.
When following the build instructions before this change then trying
to run the RPC tests, a unix user would get an error when python
tried to import zmq.
There may be other dependencies that should be added to the docs,
particularly ones for non-unix systems. This is the only unlisted
dependency I encountered using linux.
In rpc-tests.py, don't override BITCOIND and BITCOINCLI if they're
already set. Makes it possible to run the tests with either another tree
or the GUI.
Thanks to @MarcoFalke @dexX7 @laanwj for review.
Zcash: Just the --coverage flag for rpc-tests.py, as we backported the
rest of the coverage backend in zcash/zcash#4411.
1) created rpc-tests.py
2) deleted rpc-tests.sh
3) travis.yml points to rpc-tests.py
4) Modified Makefile.am
5) Updated README.md
6) Added tests_config.py and deleted tests-config.sh
7) Modified configure.ac with script to set correct path in tests_config.py
Zcash: Migrated our test list over, and other necessary modifications.
The UI changes were not migrated.
Since the introduction of the ScriptForMining callback, the mining
functions (setgenerate and generate) crash with an assertion failure
(due to a NULL pointer script returned) if the keypool is empty. Fix
this by giving a proper error.
Zcash: Adapted to our MinerAddress type.
Co-authored-by: Jack Grigg <jack@electriccoin.co>
test: Use default shielded address in RPC tests where the type is irrelevant
This means we are running these tests against the recommended functionality,
and that these RPC tests will now use the faster Sapling addresses.
Improving asyncOp_sendmany redundancies (first round)
Work over async sendmany operation flow. Specifically on the utxo finding, utxo selection, total amount calculation and the tx inputs appending process. Removed several unneeded for loops over the entire utxo set, re structuring and cleaning the flow.
These are text files but their encoding does not depend on the locale.
Not all of them require utf8 but it is better to fix it at something
to remove potential unpredictability.
This is necessary on FreeBSD where no locale is set by default,
and apparently Python defaults not only the terminal encoding to the locale
but that of every text file. So without LOCALE environment it defaults text
file encoding to ASCII. This causes problems with e.g. `bitcoin.conf`.
Luckily the locale doesn't affect the default encoding for str.encode() and
bytes.decode() on Python 3, so this is the only change necessary.
allow getaddressutxos if -lightwalletd
The wallets need to be able to retrieve, via lightwalletd, the UTXO set for a given address, see https://github.com/zcash/lightwalletd/issues/312. The `getaddressutxos` rpc that was added for Insight Explorer is perfect for this, but this rpc is disabled when only `-lightwalletd` is enabled (that is, it requires `-insightexplorer`, which is a superset of `-lightwalletd`). However, the `DB_ADDRESSUNSPENTINDEX` index that supports this rpc is present when only `-lightwalletd` is enabled. This was probably a (fortuitous, as it turns out) oversight. So all that's required is to allow this rpc when `-lightwalletd` is enabled.
We don't currently have a need for newer releases, and will likely be
pinning Rust for longer periods of time once we are also pinning Clang.
Rust releases occur every six weeks, so we can pre-emptively postpone
releases through the end of this year.
- The old patch is no longer necessary because of this upstream fix:
https://github.com/boostorg/build/pull/560
- Boost 1.72 removed a <deque> from an include, which exposed a missing
include in src/httpserver.cpp.
- Boost 1.73 moved function placeholders into the boost::placeholders
namespace.
- The new patch is a fix from just after Boost 1.74 was released, fixing
a warning that was missed.
Small httpserver.cpp backports
Also includes a change to the `uiInterface.NotifyBlockTip` signal API.
These remove merge conflicts from subsequent backports for `sync.h`.
Cherry-picked from the following upstream PRs:
- bitcoin/bitcoin#6859
- bitcoin/bitcoin#7112
- Only the non-QT changes.
- bitcoin/bitcoin#7966
- bitcoin/bitcoin#8421
- We already backported the second commit in zcash/zcash#2555
Address some performance regressions
@str4d identified and fixed some performance regressions in our elliptic curve and proving crates, and we also changed to complete formulas in `bls12_381` to get some better performance in some cases. The result:
```
v3.1.0
"runningtime": 0.59883
before upgrading rust:
"runningtime": 0.823932
after upgrading rust:
"runningtime": 0.83004
after upgrading rust packages:
"runningtime": 0.763065
```
which gets us some of the way back to where we were.
A few "a->an" and "an->a".
"Shows, if the supplied default SOCKS5 proxy" -> "Shows if the supplied default SOCKS5 proxy". Change made on 3 occurrences.
"without fully understanding the ramification of a command" -> "without fully understanding the ramifications of a command".
Removed duplicate words such as "the the".
Zcash: Only the changes to files and code that we have.
wallet: Add ANY_TADDR special string to z_sendmany
When using this special string as the from address, non-coinbase UTXOs
from any transparent addresses within the wallet will be used to fund the
transaction. Change outputs will be sent to a new transparent address,
as with any other spend of transparent funds.
Closeszcash/zcash#3640.
depends: Switch to `cargo vendor` for Rust dependencies
When we first integrated Rust into our build system, we had two
limitations:
- We were building the `librustzcash` FFI library as a dependency, and
therefore needed access to its crate dependencies in the depends
system.
- Gitian builds happen offline, so we needed to fetch any crate
dependencies ahead of time, and then configure cargo to use these in
an offline environment.
At the time, `cargo` already had support for "Source Replacement", but
there was no easy way to package the dependencies in the necessary way.
What we implemented was effectively the `cargo-vendor` tool, built using
Makefiles. A noticeable downside was that we were pinning dependencies
twice: once in the `Cargo.lock` for the FFI library, and again in our
depends system.
Since then, `cargo-vendor` has been upstreamed into `cargo` itself, and
we have moved `librustzcash` into this repository. We can therefore use
`cargo vendor` directly from our pinned Rust compiler to fetch the
dependencies, and rely on our local `Cargo.lock` to pin the specific
crates we are relying on.
add shielded balance to getwalletinfo
Closeszcash/zcash#3939
It is based on the definition that unconfirmed balance has 0 confirmations; anything else is regular balance.
Implementation:
1. `getBalanceZaddr` uses one version of `GetFilteredNotes`, we want to use the other version that allow us to get balances inside min and max confirmations.
2. `shielded_unconfirmed_balance`, and `shielded_balance` are obtained by calling `getBalanceZaddr` with different min and max confirmations according to the definitions from above.
When using this special string as the from address, UTXOs from any
transparent addresses within the wallet will be used to fund the
transaction. Change outputs will be sent to a new transparent address,
as with any other spend of transparent funds.
Closeszcash/zcash#3640.
Use the Rust tracing crate for C++ logging
This PR swaps in the `tracing` crate (via FFI) for logging to either standard
output or `debug.log`. It transparently maps all existing `LogPrintf` and
`LogPrint` invocations to info-level `tracing` events, and passes through
correct file and line information. `error` invocations are mapped to error-level
`tracing` events, currently without line information (due to the way that
`error` is used in the codebase; swapping individual callsites to the new
`LogError` macro will provide that information).
The end-goal for this change is that we don't need to make any disruptive
changes to the codebase, but we can start to leverage `tracing`-specific
functionality where we want to, such as providing extra fields on certain log
lines (that can be filtered for), adding spans to record the flow of execution
through `zcashd`, and logging within C++ and Rust simultaneously. Support
for extra fields on spans and events will be added in a subsequent PR.
The `-debug` config options are converted at launch into their corresponding
directives for tracing's `EnvFilter`. The new `setlogfilter` RPC method allows
this filter to be reloaded dynamically. The syntax is documented in the
`setlogfilter` help text, as well as here:
https://docs.rs/tracing-subscriber/0.2.7/tracing_subscriber/filter/struct.EnvFilter.html#directives
When `-printtoconsole` is specified, the output now includes timestamps and
ANSI encoding :)
A new safe_depends() checker is added, which allows the '.' character on
top of the existing safe() characters.
The time crate is postponed to 2021-02-01, by which time hopefully the
chrono crate will have figured out what it is doing with that dependency.
flush witness cache (SetBestChain()) on clean shutdown
Closes#4596, follow-on to #4573. In addition to flushing witness data on shutdown, fix the RPC test that was preventing this change from being part of #4573.
Flush witness data to disk only when it's consistent
Closes#4301. Running this PR's code will not repair a data directory that has been affected by this problem; that requires starting zcashd with the `-rescan` or `-reindex` options.
This makes the test more deterministic. When the nodes are split, and
partition B (node 1) mines a joinsplit transaction, that block only
sometimes appears on the blockchain after the nodes are all reconnected.
ZIP212 implementation
Closes#4557.
(description by @ebfull, taken from #4575)
* The `SaplingNote` structure has a new enum called `zip212Enabled`. This
member is private and reflects whether the note was or is being created
using the derivation method of ZIP 212 (i.e., `BeforeZip212` or `AfterZip212`).
* The `SaplingNotePlaintext` structure has a new unsigned char member
`leadbyte`. This member is private and contains the leading byte of the
plaintext (e.g. `0x01`, `0x02`).
* The serialization of `SaplingNotePlaintext` sets `zip212Enabled` to
`BeforeZip212` iff the serialized note plaintext version is not `0x01`.
* The `r`/`rcm` fields have been removed and replaced with a private field
`rseed`. `SaplingNote` and `SaplingNotePlaintext` now have a helper method
`rcm()` which returns the `rcm` either by deriving it with `rseed`
(if `zip212Enabled` is `AfterZip212`) or returning `rseed` by interpreting
`rseed` as `rcm`.
* All the methods of obtaining a `SaplingNote` account for these changes:
- The `SaplingNote` constructor that is used by e.g. the transaction builder,
and internally samples random `rcm`, now takes a `zip212Enabled` argument
to decide whether to sample `rcm` the "old" way or the "new" way.
- The bare constructor for `SaplingNote` is removed.
- The other constructor which takes the raw contents of the note is only used
in tests or in `Note.cpp`, but now also takes a `zip212Enabled` argument.
- The other way of obtaining a note, by calling `SaplingNotePlaintext::note()`,
has been adjusted.
* The `SaplingNotePlaintext` class now has an `generate_or_derive_esk()` method
that either samples a random `esk` or derives it using the local `rseed`
depending on the value of `leadbyte`.
* The encryption routine is modified to consult `generate_or_derive_esk()` and
provide it to the note encryption object.
* The note encryption objects now take an optional `esk` as input and otherwise
sample a random `esk` internally. This API functionality is preserved to allow
for testing.
* The `SaplingNotePlaintext` decryption routines are modified:
- The out and enc decryption routines now check that `epk` is consistent with
the derived `esk`.
- The out decryption routine for plaintexts also checks that `esk` is
consistent with what is derived by the note.
* The miner and transaction builder consult the activation of Canopy when
creating `SaplingNote`s.
* The consensus rules are modified so that shielded outputs (miner rewards)
must have `v2` note plaintexts after Canopy has activated.
[ZIP 211] Disabling Addition of New Value to the Sprout Value Pool
Disables Sprout outputs after NU4 by checking for nonzero `vpub_old` in transactions after NU4 activation height.
Adds gtests to check expected behaviour before and after NU4 activation height.
edit:
Also modifies `z_` methods in `rpcwallet`, and adds a matching RPC test.
Implements [ZIP 211](https://zips.z.cash/zip-0211), closes#4479
Add funding streams to consensus parameters.
Add funding stream payments to coinbase txns generated by the miner.
* Reduce valueBalance for shielded outputs to funding streams.
* Ensure we produce binding signatures in any case where shielded
outputs go to either a funding stream or the miner.
Use the cached consensusBranchId in DisconnectBlock
If a node is started with a set of network upgrades that don't match the
serialized chain (such as when we implement NU rollbacks on testnet),
RewindBlockIndex will disconnect each block in the chain until it
reaches the most recent block that agrees with the node's set of network
upgrades. However, the blocks themselves should be disconnected using
the consensus branch ID that they were connected with, which is
persisted alongside the chain and reconstructed in LoadBlockIndex.
Add confirmations, blockheight, blockindex and blocktime to z_listreceivedbyaddress
Fixes https://github.com/zcash/zcash/issues/3724
1- There was a PR to add confirmations to this call at https://github.com/zcash/zcash/pull/3836
I ported the commit from there and fixed test case by incrementing the confirmations as suggested at: https://github.com/zcash/zcash/pull/3836#issuecomment-499927807
2- Then added `blockheight`, `blockindex` and `blocktime`. To avoid some duplicated code (Sprout/Sapling) created a structure `trxblock`.
3- Original issue requests only time and blockindex however i think height is also important; if `blockindex` is the position of the transaction in the block then you are going to need also `height` to find it.
The mempool timestamps are local to each node, and if the testing
machine is under load, they can potentially differ by a second.
Closeszcash/zcash#4439.
Co-authored-by: Ying Tong <yingtong@ethereum.org>
Add -lightwalletd experimental option
Similar to `-insightexplorer` but loading less indexes.
After testing and code review this should be able to close https://github.com/zcash/zcash/issues/4326
The previous version of full_test_suite.py directly called the test
binary, which was being compiled at the same time as the static library.
However, by passing the --tests argument to cargo, rustc was ignoring
several important release-profile configurations, and was also
attempting to link the test binary, which was breaking cross-compilation
builds.
This commit alters src/Makefile.am to only build the static library, and
leaves test compilation to the test runner itself. This ensures that the
tests are only compiled for native builds, when the tests will be run on
the same platform.
Add support for Sapling full viewing keys
This PR adds Sapling support to `z_exportviewingkey` and `z_importviewingkey`, and stores imported Sapling viewing keys in the wallet.
Closes#3060.
Replace the `bitcoin-cli -rpcwait` after spawning bitcoind
with our own loop that detects when bitcoind exits prematurely.
And if one node fails to start, stop the others.
This prevents a hang in such a case (see #7463).
Thanks to @MarcoFalke @dexX7 @laanwj for review.
Zcash: Includes all the backend changes, but excludes the --coverage flag
added to rpc-tests.py as we have not yet migrated away from rpc-tests.sh.
Additional librustzcash integration
This adds librustzcash tests to the full test suite, and brings in the release profile configurations that are currently present in the librustzcash workspace on the other repository. It's very important that we build librustzcash with panic=abort because otherwise the unwinding panics across FFI boundaries could cause undefined behavior.
Allow negative heights in RPC calls
For issue https://github.com/zcash/zcash/issues/2197
Currently adding the feature to `getblock` and `getblockhash`. There is another candidate: `getblocksubsidy` however i want to have some review about these 2 first before repeating what could be a bad approach.
z_viewtransaction
This RPC method returns all decryptable information for any transaction in the wallet.
Several values are conditionally included in the output for convenience:
- `recovered`: True if an output is not for a Sapling address in the wallet.
- `memoStr`: The text form of an output's memo, if it is valid UTF-8.
- Values are provided both in decimal currency units, and integer zatoshis.
Jack Grigg
Steven Smith
Homu
therealyingtong
Kris Nuttycombe
Jonas Schnelli
MarcoFalke
Suhas Daftuar
Jonas Schnelli
Taylor Hornby
Alfredo Garcia
Daira Hopwood
Dimitris Apostolou
Josh Ellithorpe
John Newbery
fanquake
Alex Morcos
isle2983
whythat
Cory Fields
Suhas Daftuar
Elliot Olds
Wladimir J. van der Laan
James O'Beirne
Jeff Garzik
ptschip
Casey Rodarmor
Daniel Kraft
Jesse Cohen
mdr0id
furszy
Sean Bowe
Larry Ruane
Dimitris Tsapakidis
zebambam
therealyingtong
Kris Nuttycombe
therealyingtong
Rod Vagg
Anthony Towns
Eirik Ogilvie-Wigley
NicolasDorier
Marshall Gaucher