All the text from a make action is passed as arguments to a single
execve call, and it can't be longer than the maximum size allowed by the
operating system. We now have enough Rust crates vendored by the depends
system that we are hitting this limit here.
Use the Rust tracing crate for C++ logging
This PR swaps in the `tracing` crate (via FFI) for logging to either standard
output or `debug.log`. It transparently maps all existing `LogPrintf` and
`LogPrint` invocations to info-level `tracing` events, and passes through
correct file and line information. `error` invocations are mapped to error-level
`tracing` events, currently without line information (due to the way that
`error` is used in the codebase; swapping individual callsites to the new
`LogError` macro will provide that information).
The end-goal for this change is that we don't need to make any disruptive
changes to the codebase, but we can start to leverage `tracing`-specific
functionality where we want to, such as providing extra fields on certain log
lines (that can be filtered for), adding spans to record the flow of execution
through `zcashd`, and logging within C++ and Rust simultaneously. Support
for extra fields on spans and events will be added in a subsequent PR.
The `-debug` config options are converted at launch into their corresponding
directives for tracing's `EnvFilter`. The new `setlogfilter` RPC method allows
this filter to be reloaded dynamically. The syntax is documented in the
`setlogfilter` help text, as well as here:
https://docs.rs/tracing-subscriber/0.2.7/tracing_subscriber/filter/struct.EnvFilter.html#directives
When `-printtoconsole` is specified, the output now includes timestamps and
ANSI encoding :)
depends: Revert to using upstreams as primary download paths
We use the depends system for vendoring `zcashd` dependencies, pinning them
with SHA-256 hashes. It supports fetching dependencies from both their
upstream archive source, and a mirror operated by ECC.
In #816, we switched to the ECC mirror as the primary source, due to an
unreliable upstream (SourceForge). However, this only addressed the symptom
(that dependency builds would reliably fail with an unreliable upstream that
was serving incorrect files). In particular, if the ECC mirror were to become
similarly unreliable, the issue would return.
This PR fixes the core problem, by downloading dependencies and checking
their hashes as an atomic operation. This gives us greater resiliency, as
both the primary and fallback would need to fail in order to halt the build.
Having addressed this problem, we also switch back to using upstreams as
primary download paths.
Build BDB utilities
To install the binaries we need to build with just `install` instead of `install_lib` and `install_include`, this will install everything.
Then the binaries will be moved to a folder in `zcutil` directory. We can just leave them in staging however the user might have a hard time to find them there.
Closes https://github.com/zcash/zcash/issues/4537
The previous behaviour was to use FALLBACK_DOWNLOAD_PATH to download
dependencies if the primary did not resolve. This was not resilient
against primaries that either mis-report HTTP status codes (e.g.
SourceForge returning 200 OK alongside a 404 webpage), or did not
guarantee artifacts to be bit-stable (e.g. GitHub regenerating commit
archive caches in a non-reproducible manner); in either case, the
incorrect file would be fetched and then the build would fail due to
hash mismatch.
The new behaviour is to download dependencies and check their hashes as
an atomic operation, and use FALLBACK_DOWNLOAD_PATH if any part of the
operation fails.
Note that this does not _enable_ lto by default in any way, only hooks up the
machinery for -flto to work correctly.
enable-lto-support is explicitly used for pinned-clang because we know it
works. It is neither enabled nor disabled in the external clang case so that
it can be auto-detected.
For depends builds this was fixed by fbcfcf69, which deleted the conflicting
headers. When we no longer control the clang installation, we need to ensure
that the SDK's libc++ headers are used rather than the ones shipped with clang.
We can do that by turning off the default include path and hard-coding our own.
This hard-coded path is ok because we control (via SDK packaging) where these
headers end-up.
Side-note: Now that this path is hard-coded in depends, we can potentially
package the SDK differently, as the c++ folder can live wherever is most
convenient for us.
Bring the librustzcash crate into this repository
Rust dependencies are now canonically pinned within this repository by
`Cargo.lock`. We continue to use the depends system for vendoring the
dependencies, to ensure our Gitian builds continue to function (which have
no network access at build time, and fetch dependencies separately).
The `--enable-online-rust` configure flag replicates the behaviour of the
`LIBRUSTZCASH_OVERRIDE` environment variable (enabling the build system to
use https://crates.io instead of vendored dependencies).
This pulls in the exact version of `librustzcash` that we currently depend on
(corresponding to the `0.1.0` tag in https://github.com/zcash/librustzcash).
The changes to the crate since then will be pulled in as a separate PR.
Part of zcash/librustzcash#155.
Part of #4230.
Upgrade libsodium to 1.0.18
Includes patches that maintain consensus compatibility with libsodium 1.0.15 for Ed25519 pubkey and signature validation.
Replaces #4239. Closes#2872.
Jack Grigg
Homu
Kris Nuttycombe
Cory Fields
Carl Dong
fanquake
Alfredo Garcia
Sean Bowe
ca333