Commit Graph

48 Commits

Author SHA1 Message Date
teor cee0e86190 Increase the open file limit on unix platforms
If the limit is less than the ideal, try to increase it to the ideal.
If that doesn't work, try to increase the limit as high as possible.
If the limit is still less than the minimum, panic.
2020-12-02 15:32:36 +10:00
teor 741c44cd55 Implement mean_target_difficulty
And enough stub code to actually run it on the context.
2020-12-01 07:27:30 +10:00
teor 939c2b97a6 Implement AdjustedDifficulty creation
Also:
* call the difficulty check from `block_is_contextually_valid`
* add a stub `difficulty_threshold_is_valid` function
2020-12-01 07:27:30 +10:00
Henry de Valence 342eb166ff state: track UTXO provenance
This commit changes the state system and database format to track the
provenance of UTXOs, in addition to the outputs themselves.
Specifically, it tracks the following additional metadata:

- the height at which the UTXO was created;
- whether or not the UTXO was created from a coinbase transaction or
  not.

This metadata will allow us to:

- check the coinbase maturity consensus rule;
- check the coinbase inputs => no transparent outputs rule;
- implement lookup of transactions by utxo (using the height to find the
  block and then scanning the block) for a future RPC mechanism.

Closes #1342
2020-11-23 22:18:43 -08:00
Henry de Valence add94c1c45 deps: move to tokio 0.3, tower 0.4
This change is mostly mechanical, with the exception of the changes to the
`tower-batch` middleware.  This middleware was adapted from `tower::buffer`,
and the `tower::buffer` code was changed to implement its own bounded queue,
because Tokio 0.3 removed the `mpsc::Sender::poll_send` method.  See

ddc64e8d4d

for more context on the Tower changes.  To match Tower as closely as possible
in order to be able to upstream `tower-batch`, those changes are copied from
`tower::Buffer` to `tower-batch`.
2020-11-20 10:08:16 -08:00
teor ec00ee4cf0
Stop using /dev/shm on Linux (#1338)
Some systems have a very small /dev/shm, for example, see:
https://github.com/docker-library/postgres/issues/416

So we should just use the temporary directory on all operating systems.

Also:
* use TempDir to generate the temporary path
* delete the code that we copied from sled
* prefix the temporary path with the state version and network
2020-11-20 13:01:19 +10:00
Jane Lusby 4c9bb87df2
zebra-state: replace sled with rocksdb (#1325)
## Motivation

Prior to this PR we've been using `sled` as our database for storing persistent chain data on the disk between boots. We picked sled over rocksdb to minimize our c++ dependencies despite it being a less mature codebase. The theory was if it worked well enough we'd prefer to have a pure rust codebase, but if we ever ran into problems we knew we could easily swap it out with rocksdb.

Well, we ran into problems. Sled's memory usage was particularly high, and it seemed to be leaking memory. On top of all that, the performance for writes was pretty poor, causing us to become bottle-necked on sled instead of the network.

## Solution

This PR replaces `sled` with `rocksdb`. We've seen a 10x improvement in memory usage out of the box, no more leaking, and much better write performance. With this change writing chain data to disk is no longer a limiting factor in how quickly we can sync the chain.

The code in this pull request has:
  - [x] Documentation Comments
  - [x] Unit Tests and Property Tests

## Review

@hdevalence
2020-11-18 18:05:06 -08:00
dependabot[bot] 18eb4d1c31 build(deps): bump primitive-types from 0.7.2 to 0.7.3
Bumps [primitive-types](https://github.com/paritytech/parity-common) from 0.7.2 to 0.7.3.
- [Release notes](https://github.com/paritytech/parity-common/releases)
- [Commits](https://github.com/paritytech/parity-common/compare/primitive-types-v0.7.2...primitive-types-v0.7.3)

Signed-off-by: dependabot[bot] <support@github.com>
2020-11-13 15:59:04 -05:00
dependabot[bot] 8c5f6d0177 build(deps): bump once_cell from 1.5.1 to 1.5.2
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.5.1 to 1.5.2.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.5.1...v1.5.2)

Signed-off-by: dependabot[bot] <support@github.com>
2020-11-13 14:48:11 -05:00
Jane Lusby 1bc833dbcc Move work conversion helpers to test code 2020-11-12 09:14:52 -05:00
dependabot[bot] a7ad73feac build(deps): bump sled from 0.34.4 to 0.34.5
Bumps [sled](https://github.com/spacejam/sled) from 0.34.4 to 0.34.5.
- [Release notes](https://github.com/spacejam/sled/releases)
- [Changelog](https://github.com/spacejam/sled/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spacejam/sled/compare/v0.34.4...v0.34.5)

Signed-off-by: dependabot[bot] <support@github.com>
2020-11-10 11:01:06 -05:00
dependabot[bot] e87340594a build(deps): bump thiserror from 1.0.21 to 1.0.22
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.21 to 1.0.22.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.21...1.0.22)

Signed-off-by: dependabot[bot] <support@github.com>
2020-11-04 19:28:42 -05:00
dependabot[bot] 83c844abb5 build(deps): bump futures from 0.3.6 to 0.3.7
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.6 to 0.3.7.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.6...0.3.7)

Signed-off-by: dependabot[bot] <support@github.com>
2020-10-27 02:28:48 -04:00
Jane Lusby 855f9b5bcb
Implement MVP of NonFinalizedState and integrate it with the state service (#1101)
* implement most of the chain functions
* implement fork
* fix outpoint handling in Chain struct
* update expect for work
* split utxo into two sets
* update the Chain definition
* remove allow attribute in zebra-state/lib.rs
* merge ChainSet type into MemoryState
* Add error messages to asserts
* export proptest impls for use in downstream crates
* add testjob for disabled feature in zebra-chain
* try to fix github actions syntax
* add module doc comment
* update RFC for utxos
* add missing header
* working proptest for Chain
* propagate back results over channel
* Start updating RFC to match changes
* implement queued block pruning
* and now it syncs wooo!
* remove empty modules
* setup config for proptests
* re-enable missing_docs lint
* update RFC to match changes in impl
* add documentation
* use more explicit variable names
2020-10-08 13:07:32 +10:00
dependabot[bot] 1b7bf61f96 build(deps): bump thiserror from 1.0.20 to 1.0.21
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.20 to 1.0.21.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.20...1.0.21)

Signed-off-by: dependabot[bot] <support@github.com>
2020-10-07 18:25:33 -04:00
dependabot[bot] 08f363e1cb build(deps): bump futures from 0.3.5 to 0.3.6
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.5 to 0.3.6.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.5...0.3.6)

Signed-off-by: dependabot[bot] <support@github.com>
2020-10-06 16:54:08 -04:00
Jane Lusby 86ed13060f
Add tests for `Chain` implementation (#1093)
* Begin work on RFC5 implementation

* I think this is necessary

* holy shit supertrait implemented via subtrait

* implement most of the chain functions

* change to slightly better name

* implement fork

* fix outpoint handling in Chain struct

* update expect for work

* resolve review comment

* split utxo into two sets

* update the Chain definition

* just a little more

* update comment

* Apply suggestions from code review

Co-authored-by: teor <teor@riseup.net>

* apply changes from code review

* remove allow attribute in zebra-state/lib.rs

* Update zebra-state/src/memory_state.rs

Co-authored-by: teor <teor@riseup.net>

* merge ChainSet type into MemoryState

* rename state impl types

* Add error messages to asserts

* checkpoint so I can split off arbitrary changes into a PR

* export proptest impls for use in downstream crates

* add testjob for disabled feature in zebra-chain

* run rustfmt

* try to fix github actions syntax

* differentiate name

* prove that github action tests zebra-chain build without features

* revert change from last commit now that test is running

* remove accidentally introduced newline

* checkpoint

* add module doc comment

* update RFC for utxos

* add missing header

* working proptest for Chain

* apply change from chain impl PR

* setup config for proptests

* Update zebra-chain/src/block/arbitrary.rs

Co-authored-by: teor <teor@riseup.net>

* run rustfmt

Co-authored-by: teor <teor@riseup.net>
2020-10-02 15:51:51 -07:00
Jane Lusby 0b4e974c9e
export proptest impls for use in downstream crates (#1092)
* export proptest impls for use in downstream crates

* add testjob for disabled feature in zebra-chain

* run rustfmt

* try to fix github actions syntax

* differentiate name

* prove that github action tests zebra-chain build without features

* revert change from last commit now that test is running

* remove accidentally introduced newline

* Update .github/workflows/ci.yml

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>
2020-09-23 18:52:52 -07:00
Henry de Valence 6dd7318d3b deps: use Tower 0.4 from git instead of 0.3.1.
This addresses at least three pain points:

- we were affected by bugs that were already fixed in git, but not in
  the released crate;
- we can use service combinators to transform requests and responses;
- we can use the hedge middleware.

The version in git is still marked as 0.3.1 but these changes will be
part of tower 0.4: https://github.com/tower-rs/tower/issues/431
2020-09-21 14:16:56 -07:00
Henry de Valence 5d8decd224 state: process CommitFinalizedBlock out of order 2020-09-11 13:37:49 -07:00
dependabot[bot] 69032e1007 build(deps): bump sled from 0.34.3 to 0.34.4
Bumps [sled](https://github.com/spacejam/sled) from 0.34.3 to 0.34.4.
- [Release notes](https://github.com/spacejam/sled/releases)
- [Changelog](https://github.com/spacejam/sled/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spacejam/sled/compare/v0.34.3...v0.34.4)

Signed-off-by: dependabot[bot] <support@github.com>
2020-09-11 13:08:16 -04:00
teor d3b6a73f7b Improve zebra-state logging and metrics 2020-09-02 12:09:50 -04:00
dependabot[bot] 619c499ba6 build(deps): bump sled from 0.34.2 to 0.34.3
Bumps [sled](https://github.com/spacejam/sled) from 0.34.2 to 0.34.3.
- [Release notes](https://github.com/spacejam/sled/releases)
- [Changelog](https://github.com/spacejam/sled/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spacejam/sled/commits)

Signed-off-by: dependabot[bot] <support@github.com>
2020-08-24 12:49:10 -04:00
teor 5afa24588a fix: Remove unused dependencies 2020-08-20 14:49:17 -04:00
Alfredo Garcia f2d7bb3177
Command execution tests (#690)
* add zebrad acceptance tests
* add custom command test helpers that work with kill
* add and use info event for start and seed commands
* combine conflicting tests into one test case

Co-authored-by: Jane Lusby <jane@zfnd.org>
2020-08-01 16:15:26 +10:00
Jane Lusby 1015db25a8
remap the state storage to only store blocks once (#741) 2020-07-31 09:21:54 +10:00
dependabot[bot] ce7f3e6c01 build(deps): bump sled from 0.34.0 to 0.34.2
Bumps [sled](https://github.com/spacejam/sled) from 0.34.0 to 0.34.2.
- [Release notes](https://github.com/spacejam/sled/releases)
- [Changelog](https://github.com/spacejam/sled/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spacejam/sled/compare/v0.34...v0.34.2)

Signed-off-by: dependabot[bot] <support@github.com>
2020-07-29 12:27:39 -04:00
Henry de Valence 4aa00ad216 Align crate versions and user-agent with NU numbers.
We had a brief discussion on discord and it seemed like we had consensus on the
following versioning policy:

* zebrad: match major version to NU version, so we will start by releasing
  zebrad 3.0.0;

* zebra-* libraries: start by matching zebrad's version, then increment major
  versions of each library as we need to make breaking changes (potentially
  faster than the zebrad version, always respecting semver but making no
  guarantees about the longevity of major releases).

This commit sets all of the crate versions to 3.0.0-alpha.0 -- the -alpha.0
marks it as a prerelease not subject to perfect adherence to compatibility
guarantees.
2020-07-24 11:46:37 -07:00
teor 2acfcf3a90
Make the CheckpointVerifier handle partial restarts (#736)
Also put generic bounds on the BlockVerifier struct,
so we get better compilation errors.
2020-07-24 11:47:48 +10:00
dependabot[bot] e99e9653eb
build(deps): bump sled from 0.33.0 to 0.34.0 (#735)
Bumps [sled](https://github.com/spacejam/sled) from 0.33.0 to 0.34.0.
- [Release notes](https://github.com/spacejam/sled/releases)
- [Changelog](https://github.com/spacejam/sled/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spacejam/sled/compare/v0.33...v0.34)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-07-23 10:55:34 -07:00
Jane Lusby c1a1493159
use dirs crate for default location of state and config (#714)
* use dirs crate for default location of state and config
* panic if a path isn't specified for zebra-state
2020-07-23 21:12:20 +10:00
Henry de Valence c2c2a28e8b Improve tracing output in chain verifier 2020-07-22 21:59:00 -07:00
dependabot[bot] 7067ac6e0d build(deps): bump tokio from 0.2.21 to 0.2.22
Bumps [tokio](https://github.com/tokio-rs/tokio) from 0.2.21 to 0.2.22.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-0.2.21...tokio-0.2.22)

Signed-off-by: dependabot[bot] <support@github.com>
2020-07-22 12:43:38 -04:00
dependabot[bot] f70f3b324c Bump sled from 0.32.1 to 0.33.0
Bumps [sled](https://github.com/spacejam/sled) from 0.32.1 to 0.33.0.
- [Release notes](https://github.com/spacejam/sled/releases)
- [Changelog](https://github.com/spacejam/sled/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spacejam/sled/compare/v0.32.1...v0.33)

Signed-off-by: dependabot[bot] <support@github.com>
2020-07-15 10:21:59 -07:00
dependabot[bot] ebcf6b0385 Bump sled from 0.32.0 to 0.32.1
Bumps [sled](https://github.com/spacejam/sled) from 0.32.0 to 0.32.1.
- [Release notes](https://github.com/spacejam/sled/releases)
- [Changelog](https://github.com/spacejam/sled/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spacejam/sled/compare/v0.32...v0.32.1)

Signed-off-by: dependabot[bot] <support@github.com>
2020-07-13 11:51:35 -04:00
Jane Lusby 765e1e61e7
fix spandoc instrumentation for await expressions (#618)
* fix spandoc instrumentation for await expressions

* depend on released version of spandoc
2020-07-09 17:43:44 -04:00
dependabot[bot] d627889104 Bump tracing from 0.1.15 to 0.1.16
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.15 to 0.1.16.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.15...tracing-0.1.16)

Signed-off-by: dependabot[bot] <support@github.com>
2020-07-09 10:22:27 -07:00
Jane Lusby 2abf4b93a8 consolidate test init functions into zebra-test (#541)
* consolidate test init logic into one place

* rustfmt

Co-authored-by: Jane Lusby <jane@zfnd.org>
2020-06-24 11:47:18 -07:00
Henry de Valence 87a8d328d5 Rewrite zebra-state tests to use transcripts.
This makes adding more tests easier.
2020-06-24 11:47:18 -07:00
Henry de Valence 70241d3cad Fix broken git dependencies.
Pinning hashes means these won't break again in the future; they can always be updated.
2020-06-22 20:23:02 -07:00
Henry de Valence e8561d8f9e rename zebra-test-vectors to zebra-test 2020-06-22 20:23:02 -07:00
Jane Lusby 246e7cd2a9
Start testing out new version of `eyre` and `color-eyre` in zebra (#526)
* port to new version of eyre without generics

* correctly setup color_eyre hooks

Co-authored-by: Jane Lusby <jane@zfnd.org>
2020-06-22 15:36:23 -07:00
dependabot[bot] f301de41fa Bump tracing-subscriber from 0.2.5 to 0.2.6
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.5 to 0.2.6.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber/-0.2.5...tracing-subscriber-0.2.6)

Signed-off-by: dependabot[bot] <support@github.com>
2020-06-22 12:01:47 -04:00
dependabot[bot] e3ad9737e7 Bump sled from 0.31.0 to 0.32.0
Bumps [sled](https://github.com/spacejam/sled) from 0.31.0 to 0.32.0.
- [Release notes](https://github.com/spacejam/sled/releases)
- [Changelog](https://github.com/spacejam/sled/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spacejam/sled/compare/v0.31...v0.32)

Signed-off-by: dependabot[bot] <support@github.com>
2020-06-22 12:00:54 -04:00
Jane Lusby bb0553fab6
implement initial persistent state backend based on `sled` (#473) 2020-06-15 14:41:26 -07:00
Jane Lusby ac76d75813
Make blocks queryable by height (#422) 2020-06-04 10:17:49 -07:00
dependabot-preview[bot] 07b7c711fb Bump color-eyre from 0.3.2 to 0.3.4
Bumps [color-eyre](https://github.com/yaahc/color-eyre) from 0.3.2 to 0.3.4.
- [Release notes](https://github.com/yaahc/color-eyre/releases)
- [Commits](https://github.com/yaahc/color-eyre/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-06-02 22:55:02 -04:00
Jane Lusby e9af80b875
Add initial version of zebra-state (#414)
* rename zebra-storage to zebra-state

* Setup initial skeleton for zebra-state

* add test

* Apply suggestions from code review

Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>

* move shared test vectors to a common crate

Co-authored-by: Jane Lusby <jane@zfnd.org>
Co-authored-by: Henry de Valence <hdevalence@hdevalence.ca>
2020-06-02 16:16:17 -07:00