During Gitian builds, SOURCES_PATH is set to a path within the Gitian
cache. Normally this path is created as part of creating a particular
dependency's source directory, but in some situations we may vendor Rust
crates before this folder exists.
When we first integrated Rust into our build system, we had two
limitations:
- We were building the `librustzcash` FFI library as a dependency, and
therefore needed access to its crate dependencies in the depends
system.
- Gitian builds happen offline, so we needed to fetch any crate
dependencies ahead of time, and then configure cargo to use these in
an offline environment.
At the time, `cargo` already had support for "Source Replacement", but
there was no easy way to package the dependencies in the necessary way.
What we implemented was effectively the `cargo-vendor` tool, built using
Makefiles. A noticeable downside was that we were pinning dependencies
twice: once in the `Cargo.lock` for the FFI library, and again in our
depends system.
Since then, `cargo-vendor` has been upstreamed into `cargo` itself, and
we have moved `librustzcash` into this repository. We can therefore use
`cargo vendor` directly from our pinned Rust compiler to fetch the
dependencies, and rely on our local `Cargo.lock` to pin the specific
crates we are relying on.
All the text from a make action is passed as arguments to a single
execve call, and it can't be longer than the maximum size allowed by the
operating system. We now have enough Rust crates vendored by the depends
system that we are hitting this limit here.
Use the Rust tracing crate for C++ logging
This PR swaps in the `tracing` crate (via FFI) for logging to either standard
output or `debug.log`. It transparently maps all existing `LogPrintf` and
`LogPrint` invocations to info-level `tracing` events, and passes through
correct file and line information. `error` invocations are mapped to error-level
`tracing` events, currently without line information (due to the way that
`error` is used in the codebase; swapping individual callsites to the new
`LogError` macro will provide that information).
The end-goal for this change is that we don't need to make any disruptive
changes to the codebase, but we can start to leverage `tracing`-specific
functionality where we want to, such as providing extra fields on certain log
lines (that can be filtered for), adding spans to record the flow of execution
through `zcashd`, and logging within C++ and Rust simultaneously. Support
for extra fields on spans and events will be added in a subsequent PR.
The `-debug` config options are converted at launch into their corresponding
directives for tracing's `EnvFilter`. The new `setlogfilter` RPC method allows
this filter to be reloaded dynamically. The syntax is documented in the
`setlogfilter` help text, as well as here:
https://docs.rs/tracing-subscriber/0.2.7/tracing_subscriber/filter/struct.EnvFilter.html#directives
When `-printtoconsole` is specified, the output now includes timestamps and
ANSI encoding :)
depends: Revert to using upstreams as primary download paths
We use the depends system for vendoring `zcashd` dependencies, pinning them
with SHA-256 hashes. It supports fetching dependencies from both their
upstream archive source, and a mirror operated by ECC.
In #816, we switched to the ECC mirror as the primary source, due to an
unreliable upstream (SourceForge). However, this only addressed the symptom
(that dependency builds would reliably fail with an unreliable upstream that
was serving incorrect files). In particular, if the ECC mirror were to become
similarly unreliable, the issue would return.
This PR fixes the core problem, by downloading dependencies and checking
their hashes as an atomic operation. This gives us greater resiliency, as
both the primary and fallback would need to fail in order to halt the build.
Having addressed this problem, we also switch back to using upstreams as
primary download paths.
Build BDB utilities
To install the binaries we need to build with just `install` instead of `install_lib` and `install_include`, this will install everything.
Then the binaries will be moved to a folder in `zcutil` directory. We can just leave them in staging however the user might have a hard time to find them there.
Closes https://github.com/zcash/zcash/issues/4537
The previous behaviour was to use FALLBACK_DOWNLOAD_PATH to download
dependencies if the primary did not resolve. This was not resilient
against primaries that either mis-report HTTP status codes (e.g.
SourceForge returning 200 OK alongside a 404 webpage), or did not
guarantee artifacts to be bit-stable (e.g. GitHub regenerating commit
archive caches in a non-reproducible manner); in either case, the
incorrect file would be fetched and then the build would fail due to
hash mismatch.
The new behaviour is to download dependencies and check their hashes as
an atomic operation, and use FALLBACK_DOWNLOAD_PATH if any part of the
operation fails.
Note that this does not _enable_ lto by default in any way, only hooks up the
machinery for -flto to work correctly.
enable-lto-support is explicitly used for pinned-clang because we know it
works. It is neither enabled nor disabled in the external clang case so that
it can be auto-detected.
For depends builds this was fixed by fbcfcf69, which deleted the conflicting
headers. When we no longer control the clang installation, we need to ensure
that the SDK's libc++ headers are used rather than the ones shipped with clang.
We can do that by turning off the default include path and hard-coding our own.
This hard-coded path is ok because we control (via SDK packaging) where these
headers end-up.
Side-note: Now that this path is hard-coded in depends, we can potentially
package the SDK differently, as the c++ folder can live wherever is most
convenient for us.
Bring the librustzcash crate into this repository
Rust dependencies are now canonically pinned within this repository by
`Cargo.lock`. We continue to use the depends system for vendoring the
dependencies, to ensure our Gitian builds continue to function (which have
no network access at build time, and fetch dependencies separately).
The `--enable-online-rust` configure flag replicates the behaviour of the
`LIBRUSTZCASH_OVERRIDE` environment variable (enabling the build system to
use https://crates.io instead of vendored dependencies).
This pulls in the exact version of `librustzcash` that we currently depend on
(corresponding to the `0.1.0` tag in https://github.com/zcash/librustzcash).
The changes to the crate since then will be pulled in as a separate PR.
Part of zcash/librustzcash#155.
Part of #4230.
Upgrade libsodium to 1.0.18
Includes patches that maintain consensus compatibility with libsodium 1.0.15 for Ed25519 pubkey and signature validation.
Replaces #4239. Closes#2872.
The --enable-online-rust configure flag replicates the behaviour of the
LIBRUSTZCASH_OVERRIDE environment variable (enabling the build system to
use crates.io instead of vendored dependencies).
z_viewtransaction
This RPC method returns all decryptable information for any transaction in the wallet.
Several values are conditionally included in the output for convenience:
- `recovered`: True if an output is not for a Sapling address in the wallet.
- `memoStr`: The text form of an output's memo, if it is valid UTF-8.
- Values are provided both in decimal currency units, and integer zatoshis.
This reverts commit 734e594c2c.
It appears that this was fixing a single issue with the FreeBSD build,
while b6d0996cec addressed the underlying
cause. However, the change in this commit overrides the underlying fix
when cross-compiling for Darwin.
The build environment is applied to the entire configured build_cmds
section, but it doesn't carry over past the && between the gmock and
gtest builds. This causes problems when cross-compiling for darwin, as
the correctly-configured clang++ is provided in the build environment.
This sidesteps the problem where the atomics check tries to run a test
binary, which cannot be performed during cross compilation. We should
replace this with a better solution in future.
Part of #3710.
Some dependency sources were downloaded via http, even though https (SSL/TLS) options are available.
Even if we potentially check the integrity of the downloaded files via hash comparison, we should make
use of this additional security layer.
Zcash:
native_cctools.mk
For normal users, --no-same-owner is default, but not so for root, where
it is assumed that root can change ownership willy-nilly. This is not
the case for privilege-limited container environments where we gaslight
the process into thinking it's root.
Zcash: Excludes QT changes
ld64 is threaded, and uses a worker for each CPU to parse input files. But
there's a bug in the parser causing dependencies to be calculated differently
based on which files have already been parsed.
As a result, builders with more CPUs are more likely to see non-determinism.
This looks to have been fixed in a newer version of ld64, so just disable
threading for now. There's no noticible slowdown.
clang: 3.7.1
cctools: 877.8
ld64: 253.9
Zcash: Second part of f25209a3e1e6488d4d44de15b0f113d2302e9aee from
upstream (we merged the first part in #2697).
librustzcash now requires a minimum of Rust 1.36.0.
The proc-macro2, quote, syn, and unicode-xid dependencies are pulled in
because we moved to using ff_derive inside pairing to derive the
BLS12-381 fields. We will be going back to explicit implementations with
the jubjub and bls12_381 crates, so these dependencies will disappear
once that is done.
The autocfg crate is pulled in by the upgraded num-integer crate, which
is transitively used by fpe. Rewriting fpe to not use num-bigint would
drop:
- autocfg
- num-bigint
- num-integer
- num-traits
We primarily depend on rand_core in our crates. The rand crate, and its
other dependencies, are pulled in for two reasons:
- The group crate exposes testing helper functions in its public API
that use distribution sampling APIs in the rand crate.
- zcash_primitives::transaction::Builder uses rand::seq::SliceRandom to
shuffle the order of Sapling spends and outputs.
Refactoring these in order to drop rand would additionally drop:
- c2-chacha
- rand_chacha
- rand_hc
- rand_xorshift
Per the Boost.Build documentation, user-config.jam is supposed to only
be located in the user's home directory, and this appears to be enforced
on FreeBSD.
On FreeBSD 12, the bash package is not necessarily installed to /bin/bash,
which the install script assumes is the location. Since we depend on bash
for building anyway, we can just explicitly call it.
depends: Support additional cross-compilation targets in Rust
This will make it easier for third parties to cross-compile `zcashd` for other platforms. The third commit in this PR shows how to add a new target to the Rust dependency builder.
The default Rust target during cross-compilation is the canonical host, which is derived from `HOST` using `depends/config.sub`. If the canonical host differs from the required Rust target, add the necessary mapping in addition to the target itself.
Also includes fixes for cross-compiling aarch64 targets.
This sidesteps the problem where the atomics check tries to run a test
binary, which cannot be performed during cross compilation. We should
replace this with a better solution in future.
Part of #3710.
Usage on Debian / Ubuntu:
> $ sudo apt install g++-aarch64-linux-gnu
> $ HOST=aarch64-linux-gnu ./zcutil/build.sh
Currently fails to cross-compile due to later configuration issues in
the depends system that need to be worked around.
The native binaries generated in the depends system are available on the path,
but system binaries are still visible. This change ensures we use cargo from
the depends system rather than whatever might be installed locally.
The only upstream change relative to the previous commit is that the
various Zcash-specific dependencies have been pulled into a cargo
workspace. The dependecies in the workspace use the same commits as the
crates we had previously vendored.
The patches are necessary to handle the fact that cargo requires that
dev dependencies are available even if not used, and we would otherwise
need to vendor all the underlying crates.
Make Rust compilation mandatory
The temporary integration check in CheckEquihashSolution() remains, until we
have "real" Rust code to integrate.
Closes#2688.
Add filename and SHA256 hash for Windows Rust package
When running `make download` in the `depends` directory, the `download-win` target (which `download` depends on) generates an error when it runs the commands defined in `check_or_remove_sources`:
```Makefile
sha256sum: /home/vagrant/zcash/depends/work/download/rust-1.16.0/..hash: no properly formatted SHA256 checksum lines found
funcs.mk:242: recipe for target '/home/vagrant/gitian-builder/cache/common/download-stamps/.stamp_fetched-rust-.hash' failed
make[1]: *** [/home/vagrant/gitian-builder/cache/common/download-stamps/.stamp_fetched-rust-.hash] Error 1
make[1]: Leaving directory '/home/vagrant/zcash/depends'
Makefile:153: recipe for target 'download-win' failed
make: *** [download-win] Error 2
```
The reason for the error is that `depends/packages/rust.mk` defines `rust_file_name_linux` and `rust_file_name_darwin` but leaves `rust_file_name_mingw32` undefined.
A directory of available rust downloads is here: https://static.rust-lang.org/dist/index.html
The closest windows analog in that list (using the same version number as currently defined in `rust.mk`) appears to be `rust-mingw-1.16.0-x86_64-pc-windows-gnu.tar.gz`. A corresponding sha256 value is also given in `rust-mingw-1.16.0-x86_64-pc-windows-gnu.tar.gz.sha256`.
After adding these values to `rust.mk`, the rust-mingw tar package was downloaded along with the rest of the dependencies and the above error message went away.
PATH variable containing spaces cause build failure
Spaces in PATH variable is creating build issues (observed on macOS). For example "VMware Fusion" adds itself to PATH like `/Applications/VMware Fusion.app/Contents/Public`.
Currently Travis's wget fails fetching qrencode:
Fetching qrencode...
ERROR: no certificate subject alternative name matches
requested host name `fukuchi.org'.
To connect to fukuchi.org insecurely, use `--no-check-certificate'.
OpenSSL: error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error
Unable to establish SSL connection.
make: *** [/home/travis/build/luke-jr/bitcoin/depends/sources/download-stamps/.stamp_fetched-qrencode-qrencode-3.4.4.tar.bz2.hash] Error 4
Closes#2279. Configures CMake to enable C++11, build static libaries
and only build cpp bindings with minimal dependencies. Documentation,
examples, tests and other language bindings are no longer built.
CMake will no longer try to find commands and packages which are not
required for building the target.
Debian 8 stable ships with gcc 4.9.2 and cmake 3.0.2. Previously
the depends package used CMAKE_CXX_STANDARD to tell cmake to use
C++11, but the option requires cmakes 3.1+. To resolve the issue
we now update relevant CMakeLists.txt and set CMAKE_CXX_FLAGS.
Add a patch that seems to be necessary for compatibilty of libevent
2.0.22 with recent mingw-w64 gcc versions (at least GCC 5.3.1 from Ubuntu
16.04).
Without this patch the Content-Length in the HTTP header ends up as
`Content-Length: zu`, causing communication between the RPC
client and server to break down. See discussion in #8653.
Source: https://sourceforge.net/p/levent/bugs/363/
Thanks to @sstone for the suggestion.
Sean Bowe
Jack Grigg
Homu
Kris Nuttycombe
Cory Fields
Carl Dong
fanquake
Alfredo Garcia
ca333
Dimitris Apostolou
Miles Manley
Taylor Hornby
Ulrich Kempken
Carl Dong
Eirik Ogilvie-Wigley
Larry Ruane
LongShao007
Simon
Jay Graber
Daira Hopwood
Charlie O'Keefe
rofl0r
Wladimir J. van der Laan
daniel
MarcoFalke
Luke Dashjr
syd
kozyilmaz