Notable changes =============== Wallet Performance Improvements ------------------------------- `zcashd 5.2.0` improved the performance of wallet scanning with multithreaded batched trial decryption of Sapling outputs. However, for some nodes this resulted in growing memory usage that would eventually cause an OOM abort. We have identified the cause of the growth, and made significant improvements to reduce the memory usage of the batch scanner. In addition, the batch scanner now has a memory limit of 100 MiB. `zcashd` now reports the following new metrics when `-prometheusport` is set: - (counter) `zcashd.wallet.batchscanner.outputs.scanned` - (gauge) `zcashd.wallet.batchscanner.size.transactions` - (gauge) `zcashd.wallet.batchscanner.usage.bytes` - (gauge) `zcashd.wallet.synced.block.height` RPC Interface ------------- - The `finalorchardroot` field in the `getblock` result and the `orchard.commitments.finalRoot` field in the `z_gettreestate` result have been changed to match the byte ordering used for the `orchard.anchor` field in the `getrawtransaction` result. These previously produced different hash values from the `orchard.anchor` field due to having been byte-flipped in their internal representation in zcashd. Changelog ========= Alex (1): build: update book.yml Signed-off-by: sashashura <93376818+sashashura@users.noreply.github.com> Alex Wied (2): cuckoocache: Add missing header build: Reorder link targets to properly build on Nix Andrés G. Aragoneses (1): autogen.sh: warn about needing autoconf if autoreconf is not found Ben Woosley (1): doc: Correct spelling errors in comments Conrado Gouvea (1): Include algorithm.h in cuckoocache.h Daira Hopwood (9): Add contrib/debian/copyright entry for crc32c. Apply suggestions from code review Update doc/book/src/user/security-warnings.md Update librustzcash commit and adapt to changes in `DiversifierKey`. Avoid an implicit clone of the Orchard bundle in ContextualCheckTransaction. Use prepared epks and ivks in trial decryption. Audit dependency updates. Include memory usage of the `tags` vector. This fixes *one* of the bugs pointed out by @str4d at https://github.com/zcash/zcash/pull/6156/files#r979122874 Always use a tuple as right argument of % in new Python code. DeckerSU (2): miner: fix MAXSOLS test_framework: fix AttributeError in sapling_spends_compact_digest Greg Pfeil (20): Eliminate indirection for debug log Define some basic cross-editor configuration Canonicalize some user-provided paths Add an `rpc-tests` make target Add simplejson to requirements for rpc-tests Also canonicalize paramsdir. Backport tor.md changes from readthedocs Don't recommend -reindex-chainstate. Add a finalorchardroot RPC test Fix finalorchardroot serialization Apply suggestions from code review Fix display of binary name in error messages. Address review feedback and fixed test failures Add test for `zcashd --help` Fix `zcashd --help` Apply suggestions from code review Qualify a `string` Don’t define wallet-needing code when --disable-wallet make-release.py: Versioning changes for 5.3.0. make-release.py: Updated manpages for 5.3.0. Hennadii Stepanov (5): Enable ShellCheck rules script: Lint Gitian descriptors with ShellCheck script: Enable SC2006 rule for Gitian scripts script: Enable SC2155 rule for Gitian scripts script: Enable SC2001 rule for Gitian scripts Jack Grigg (53): Squashed 'src/leveldb/' changes from f545dfabff..f8ae182c1e Squashed 'src/crc32c/' content from commit 224988680f depends: Update Rust to 1.62.1 depends: Update Clang / libcxx to LLVM 14.0.6 depends: Update Rust to 1.63.0 CI: Migrate to published versions of cargo-vet rust: Update some of the pinned dependencies rust: Add `zcash-inspect` binary for inspecting Zcash data rust: Add P2PKH signature checking to `zcash-inspect` rust: Add address inspection to `zcash-inspect` rust: Add mnemonic phrase inspection to `zcash-inspect` rust: Simplify `next_pow2` in `zcash-inspect` rust: Place tighter bound on encoded heights in `zcash-inspect` lint: Fix shell lints CI: Enforce shell lints to prevent regression lint: Fix ShellCheck lints in Zcash scripts lint: Disable some ShellChecks on Gitian descriptors rust: Add shielded sighash to `zcash-inspect` output for txs rust: `zcash-inspect` 32-byte hex as maybe a commitment or nullifier build: Build Rust library and binaries at the same time rust: Migrate to latest `zcash_primitives` revision rust: Migrate Rust tests to latest `zcash_primitives` revision wallet: Use `auto&` to avoid copying inside `ThreadNotifyWallets` wallet: Refactor `ThreadNotifyWallets` to support batch memory limits wallet: Add dynamic memory usage tracking to `BatchScanner` wallet: Collect metrics on the number of scanned outputs wallet: Set a memory limit of 100 MiB for batch scanning wallet: Only store successful trial decryptions in batch scanner depends: Update Rust to 1.64.0 qa: Postpone dependencies that require CMake qa: Postpone Clang 15 to retain LLVM 14 pin depends: Update cxx to 1.0.76 depends: Update Boost to 1.80.0 Fix clippy lints qa: Add audit policies for patched Rust crates metrics: Add gauge for the height to which the wallet is synced wallet: Move heap tracking of batch tasks behind a trait wallet: Correctly track heap usage of batch items wallet: Improve estimation of `rayon` spawned task size Squashed 'src/secp256k1/' changes from a4abaab793..efad3506a8 Squashed 'src/secp256k1/' changes from efad3506a8..1758a92ffd build: Disable secp256k1 OpenSSL tests wallet: Remove lock on cs_main from CWallet::ChainTip rust: Update to `metrics 0.20` depends: Update cxx to 1.0.78 qa: Postpone Clang 15.0.2 rust: Update to `cpufeatures 0.2.5` qa: Recommend cargo-upgrades instead of cargo-outdated rust: Audit some dependency updates rust: Update remaining dependencies make-release.py: Versioning changes for 5.3.0-rc1. make-release.py: Updated manpages for 5.3.0-rc1. make-release.py: Updated release notes and changelog for 5.3.0-rc1. James White (1): Add IPv6 support to qos.sh Kris Nuttycombe (12): Update `z_sendmany` help to clarify what happens when sending from a UA Revert "redirect and update source documentation" Fix documentation line wrapping Backport changes from zcash.readthedocs.io Move restored documentation into the zcashd book Replace manual mangement of the Sapling proving context with cxx Apply suggestions from code review Update and/or postpone dependencies for 5.3.0 Vet cxxbridge upgrade. Update to released versions of librustzcash crates. Certify librustzcash crate upgrades. Fix clippy lints. Luke Dashjr (2): Bugfix: Only use git for build info if the repository is actually the right one Bugfix: Detect genbuild.sh in repo correctly Marco Falke (1): Remove script to clean up datadirs Nathan Wilcox (1): Update rust.md Pieter Wuille (1): libsecp256k1 no longer has --with-bignum= configure option Wladimir J. van der Laan (6): build: Update build system for new leveldb doc: Add crc32c subtree to developer notes test: Add crc32c to subtree check linter test: Add crc32c exception to various linters and generation scripts build: CRC32C build system integration build: Get rid of `CLIENT_DATE` sasha (1): Make RUST_DIST in Makefile.am refer to rust-toolchain.toml (baf7d9e) user (1): README.md: Clarify distinction between protocol vs zcashd implementation; link to Zebra; line wrapping.