Add libfuzzer support
- Added libfuzzer support to the build system in zcutil.
- Added UniValue::Read fuzzer, dict, corpus.
I made a slight change to the build.sh script so as to separate dependency building vs src building, just so that I can layer the docker containers for building over in another repo to speed up the continuous build processes that will want to build multiple fuzzers from the same commit.
Closes#4548
This will make it possible to de-duplicate some components of the
tracing configuration. The layers themselves cannot be fully
de-duplicated due to type system constraints (a Layer wraps a
Subscriber, and is itself a Subscriber typed on the one it wraps).
Windows uses u16 for OS strings, requiring an owned OsString when
converting to Rust. This needs to be stored in its own variable, so we
can take an immutable reference to it in Path::new.
All the text from a make action is passed as arguments to a single
execve call, and it can't be longer than the maximum size allowed by the
operating system. We now have enough Rust crates vendored by the depends
system that we are hitting this limit here.
Remove deprecated contrib utilities
Removed a bunch of unused/deprecated utilities from `zcash/contrib`.
If you feel there are more to be removed, feel free to point them out and I will remove them as well.
Modernise macOS cross-compilation toolchain
Cherry-picked from the following upstream PRs:
- bitcoin/bitcoin#13617
- Excluding the QT GUI changes.
- bitcoin/bitcoin#17550
- bitcoin/bitcoin#16392
- Excluding the QT GUI changes.
- bitcoin/bitcoin#18589
- bitcoin/bitcoin#19240
- bitcoin/bitcoin#19407
- bitcoin/bitcoin#17919
- Only the ancillary changes, not the `FORCE_USE_SYSTEM_CLANG` change.
- bitcoin/bitcoin#19530
After these changes, macOS versions earlier than 10.12 are no longer supported.
To cross-compile for macOS:
- Follow the instructions in `contrib/macdeploy/README.md` to generate
`Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers.tar.gz`
(requires an Apple Developer Account)
- Extract it into `depends/SDKs` (creating that folder first if it does not exist)
- `HOST=x86_64-apple-darwin16 ./zcutil/build.sh`
Use the Rust tracing crate for C++ logging
This PR swaps in the `tracing` crate (via FFI) for logging to either standard
output or `debug.log`. It transparently maps all existing `LogPrintf` and
`LogPrint` invocations to info-level `tracing` events, and passes through
correct file and line information. `error` invocations are mapped to error-level
`tracing` events, currently without line information (due to the way that
`error` is used in the codebase; swapping individual callsites to the new
`LogError` macro will provide that information).
The end-goal for this change is that we don't need to make any disruptive
changes to the codebase, but we can start to leverage `tracing`-specific
functionality where we want to, such as providing extra fields on certain log
lines (that can be filtered for), adding spans to record the flow of execution
through `zcashd`, and logging within C++ and Rust simultaneously. Support
for extra fields on spans and events will be added in a subsequent PR.
The `-debug` config options are converted at launch into their corresponding
directives for tracing's `EnvFilter`. The new `setlogfilter` RPC method allows
this filter to be reloaded dynamically. The syntax is documented in the
`setlogfilter` help text, as well as here:
https://docs.rs/tracing-subscriber/0.2.7/tracing_subscriber/filter/struct.EnvFilter.html#directives
When `-printtoconsole` is specified, the output now includes timestamps and
ANSI encoding :)
depends: Revert to using upstreams as primary download paths
We use the depends system for vendoring `zcashd` dependencies, pinning them
with SHA-256 hashes. It supports fetching dependencies from both their
upstream archive source, and a mirror operated by ECC.
In #816, we switched to the ECC mirror as the primary source, due to an
unreliable upstream (SourceForge). However, this only addressed the symptom
(that dependency builds would reliably fail with an unreliable upstream that
was serving incorrect files). In particular, if the ECC mirror were to become
similarly unreliable, the issue would return.
This PR fixes the core problem, by downloading dependencies and checking
their hashes as an atomic operation. This gives us greater resiliency, as
both the primary and fallback would need to fail in order to halt the build.
Having addressed this problem, we also switch back to using upstreams as
primary download paths.
A new safe_depends() checker is added, which allows the '.' character on
top of the existing safe() characters.
The time crate is postponed to 2021-02-01, by which time hopefully the
chrono crate will have figured out what it is doing with that dependency.
Due to tracing's extensive use of the Rust type system, we have to Box
the handle in order to pass it back across the FFI. We define a
ReloadHandle trait which exposes the necessary Handle APIs.
Homu
therealyingtong
bambam
Jack Grigg
Dimitris Apostolou
elbandi