Remove deprecated contrib utilities
Removed a bunch of unused/deprecated utilities from `zcash/contrib`.
If you feel there are more to be removed, feel free to point them out and I will remove them as well.
Modernise macOS cross-compilation toolchain
Cherry-picked from the following upstream PRs:
- bitcoin/bitcoin#13617
- Excluding the QT GUI changes.
- bitcoin/bitcoin#17550
- bitcoin/bitcoin#16392
- Excluding the QT GUI changes.
- bitcoin/bitcoin#18589
- bitcoin/bitcoin#19240
- bitcoin/bitcoin#19407
- bitcoin/bitcoin#17919
- Only the ancillary changes, not the `FORCE_USE_SYSTEM_CLANG` change.
- bitcoin/bitcoin#19530
After these changes, macOS versions earlier than 10.12 are no longer supported.
To cross-compile for macOS:
- Follow the instructions in `contrib/macdeploy/README.md` to generate
`Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers.tar.gz`
(requires an Apple Developer Account)
- Extract it into `depends/SDKs` (creating that folder first if it does not exist)
- `HOST=x86_64-apple-darwin16 ./zcutil/build.sh`
Use the Rust tracing crate for C++ logging
This PR swaps in the `tracing` crate (via FFI) for logging to either standard
output or `debug.log`. It transparently maps all existing `LogPrintf` and
`LogPrint` invocations to info-level `tracing` events, and passes through
correct file and line information. `error` invocations are mapped to error-level
`tracing` events, currently without line information (due to the way that
`error` is used in the codebase; swapping individual callsites to the new
`LogError` macro will provide that information).
The end-goal for this change is that we don't need to make any disruptive
changes to the codebase, but we can start to leverage `tracing`-specific
functionality where we want to, such as providing extra fields on certain log
lines (that can be filtered for), adding spans to record the flow of execution
through `zcashd`, and logging within C++ and Rust simultaneously. Support
for extra fields on spans and events will be added in a subsequent PR.
The `-debug` config options are converted at launch into their corresponding
directives for tracing's `EnvFilter`. The new `setlogfilter` RPC method allows
this filter to be reloaded dynamically. The syntax is documented in the
`setlogfilter` help text, as well as here:
https://docs.rs/tracing-subscriber/0.2.7/tracing_subscriber/filter/struct.EnvFilter.html#directives
When `-printtoconsole` is specified, the output now includes timestamps and
ANSI encoding :)
depends: Revert to using upstreams as primary download paths
We use the depends system for vendoring `zcashd` dependencies, pinning them
with SHA-256 hashes. It supports fetching dependencies from both their
upstream archive source, and a mirror operated by ECC.
In #816, we switched to the ECC mirror as the primary source, due to an
unreliable upstream (SourceForge). However, this only addressed the symptom
(that dependency builds would reliably fail with an unreliable upstream that
was serving incorrect files). In particular, if the ECC mirror were to become
similarly unreliable, the issue would return.
This PR fixes the core problem, by downloading dependencies and checking
their hashes as an atomic operation. This gives us greater resiliency, as
both the primary and fallback would need to fail in order to halt the build.
Having addressed this problem, we also switch back to using upstreams as
primary download paths.
A new safe_depends() checker is added, which allows the '.' character on
top of the existing safe() characters.
The time crate is postponed to 2021-02-01, by which time hopefully the
chrono crate will have figured out what it is doing with that dependency.
Due to tracing's extensive use of the Rust type system, we have to Box
the handle in order to pass it back across the FFI. We define a
ReloadHandle trait which exposes the necessary Handle APIs.
The tracing crate is initialized with an optional log path, and will
either start a background thread for non-blocking log writing, or write
directly to standard output with ANSI encoding.
C preprocessor macros are used to emulate the Rust macros natively
provided by the tracing crate. They handle the creation of static
tracing callsites, and ensure that the correct file and line number
information is used for each logging site.
Refactor ProofVerifier
`ProofVerifier` was previously used to conditionally verify pre-Sapling Sprout
proofs (based on `ProofVerifier::Strict` or `ProofVerifier::Disabled` being
used), but hybrid Sprout proofs bypassed it (so were being verified multiple
times during block verification), and once `libsnark` was removed in
zcash/zcash#4060 `ProofVerifier::check` was doing nothing.
This PR refactors `ProofVerifier`, moving it out of the `libzcash` compilation
unit (so that it can depend on `primitives/transaction.h`), and moving Sprout
verification from `JSDescription::Verify` to `ProofVerifier::VerifySprout`.
Verification-skipping for Sprout proofs is re-introduced.
Additionally, the `ZCJoinSplit` global is removed from the codebase, and
`ZCJoinSplit::prove` is converted into a static function. We load the hybrid
Sprout parameters dynamically at proving time within the Rust code, and no
longer require a C++ global for any proving parameters.
As a side-effect, `libzcashconsensus.la` building with `--with-libs` is fixed,
as `primitives/transaction.cpp` no longer depends on `librustzcash.h`.
Build BDB utilities
To install the binaries we need to build with just `install` instead of `install_lib` and `install_include`, this will install everything.
Then the binaries will be moved to a folder in `zcutil` directory. We can just leave them in staging however the user might have a hard time to find them there.
Closes https://github.com/zcash/zcash/issues/4537
The previous behaviour was to use FALLBACK_DOWNLOAD_PATH to download
dependencies if the primary did not resolve. This was not resilient
against primaries that either mis-report HTTP status codes (e.g.
SourceForge returning 200 OK alongside a 404 webpage), or did not
guarantee artifacts to be bit-stable (e.g. GitHub regenerating commit
archive caches in a non-reproducible manner); in either case, the
incorrect file would be fetched and then the build would fail due to
hash mismatch.
The new behaviour is to download dependencies and check their hashes as
an atomic operation, and use FALLBACK_DOWNLOAD_PATH if any part of the
operation fails.
Check for config options that may be duplicated in the config file
`ReadConfigFile()` now enforces that only a subset of config options may be duplicated. CLI behaviour is unaltered.
Closes https://github.com/zcash/zcash/issues/4495
Replace OpenSSL AES with ctaes-based version
Backported from upstream PR https://github.com/bitcoin/bitcoin/pull/7689.
This is backported primarily to remove merge conflicts for a subsequent
backport, and also helps us towards removing OpenSSL. Its actual usage
in wallet encryption would be replaced by a more modern construction
before we make wallet encryption a supported feature, but for now this
does not affect anyone using the experimental feature.
Note that this does not _enable_ lto by default in any way, only hooks up the
machinery for -flto to work correctly.
enable-lto-support is explicitly used for pinned-clang because we know it
works. It is neither enabled nor disabled in the external clang case so that
it can be auto-detected.
Jack Grigg
Homu
Dimitris Apostolou
Daira Hopwood
teor
Taylor Hornby
Kris Nuttycombe
Cory Fields
Carl Dong