Commit Graph

941 Commits

Author SHA1 Message Date
Greg Fitzgerald a43922ccbf
Boot hashbrown (#5505)
As of Rust 1.36.0, hashbrown now implements the HashMap in std (which
implements HashSet).

https://blog.rust-lang.org/2019/07/04/Rust-1.36.0.html#a-new-hashmapk,-v%3E-implementation
2019-08-12 16:46:49 -06:00
Pankaj Garg b7a5136136
Helper functions for shreds (#5493) 2019-08-12 15:27:58 -07:00
Justin Starry 0fde19239b
Rate limit counter metrics points to one per second (#5496)
* Rate limit counter metrics points to one per second

* Remove old env var

* Test that metrics counter is incrementing

* Fix typo
2019-08-12 18:15:34 -04:00
dependabot-preview[bot] 771d1a78fd Bump libc from 0.2.60 to 0.2.61 (#5491)
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.60 to 0.2.61.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.60...0.2.61)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-08-12 15:39:26 -06:00
Sagar Dhawan a8eb0409b7
Fix gossip messages growing beyond blob size (#5460)
* fixed bloom filter math

* Add split each pull request into multiple pulls with different filters

* Rework CrdsFilter to generate all possible masks to cover the keyspace

* Limit the bloom sizes such that each pull request is no larger than mtu
2019-08-12 13:51:29 -07:00
carllin 1b84092b94
Fix slots_since_snapshot in BankForks.add_root() (#5489) 2019-08-12 11:56:03 -07:00
Pankaj Garg b1d43ace14
Add columns for data and code shreds (#5461) 2019-08-12 10:03:57 -07:00
Michael Vines 54f4d13350
Validator log filter may now be reconfigured at runtime (#5473)
* Log filter may now be reconfigured at runtime

* Add RPC API and bash script to reconfigure the log filter
2019-08-10 22:54:46 -07:00
dependabot-preview[bot] 799d3b1575 Bump nix from 0.14.1 to 0.15.0 (#5488)
Bumps [nix](https://github.com/nix-rust/nix) from 0.14.1 to 0.15.0.
- [Release notes](https://github.com/nix-rust/nix/releases)
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nix-rust/nix/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-08-10 18:48:36 -06:00
Rob Walker 5884469d11
count commitable in banking_stage (#5477) 2019-08-09 21:14:20 -07:00
Rob Walker ed093f86f9
harmonize percentage members (#5459)
* harmonize percentage members

* update tests

* update capitalization when burning fees

* verify capitalization in fee burn

* fixup
2019-08-09 13:58:46 -07:00
Michael Vines 5b4ee36cfd
Log more socket addresses at validator startup (#5471) 2019-08-08 15:38:23 -07:00
Justin Starry 4f86c0b74a
Rate limit transaction counters (#5447)
* Rate limit transaction counters

* @sakridge feedback

* Set default high metrics rate for multinode demo

* Fix tests

* Swap defaults and fix env var tests

* Only set metrics rate if not already set
2019-08-08 17:05:06 -04:00
sakridge 238aa2133d
Move local_cluster tests into own crate (#5465) 2019-08-08 11:04:33 -07:00
Michael Vines 8c15214923
Add --dev-halt-at-slot option (#5453) 2019-08-08 09:14:30 -07:00
Tyera Eulberg 5b51bb27b6
Rpc to return software version (#5456)
* Add getSoftwareVersion rpc

* Add getSoftwareVersion to doc

* Rename to getVersion and return object

* Update jsonrpc-api.md
2019-08-07 20:06:27 -06:00
Pankaj Garg 6597c71e23
Implement shred erasure recovery and reassembly (#5444)
* Implement shred erasure recovery and reassembly

* fixes and unit test

* clippy

* review comments, additional tests, and some fixes

* address review comments

* more tests and cleanup
2019-08-07 17:02:49 -07:00
carllin 8aa7a851ca
Fix hardlinking across filesystem boundaries (#5449)
* Fix hardlinking across filesystem boundaries

* create output dir for snapshot tar
2019-08-07 13:12:53 -07:00
dependabot-preview[bot] be9fa22db7 Bump hashbrown from 0.3.1 to 0.5.0 (#5450)
Bumps [hashbrown](https://github.com/rust-lang/hashbrown) from 0.3.1 to 0.5.0.
- [Release notes](https://github.com/rust-lang/hashbrown/releases)
- [Changelog](https://github.com/rust-lang/hashbrown/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/hashbrown/compare/v0.3.1...v0.5.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-08-07 08:35:54 -06:00
carllin b72c5689c9
Blow away snapshots directory on start (#5446) 2019-08-06 21:41:38 -07:00
Sagar Dhawan 9dcf3347f5 Refactor status cache and remove complex serialize/deserialize (#5335)
automerge
2019-08-06 18:47:30 -07:00
dependabot-preview[bot] 3ae9357a36 Bump hashbrown from 0.2.2 to 0.3.1 (#5381)
Bumps [hashbrown](https://github.com/rust-lang/hashbrown) from 0.2.2 to 0.3.1.
- [Release notes](https://github.com/rust-lang/hashbrown/releases)
- [Changelog](https://github.com/rust-lang/hashbrown/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/hashbrown/compare/v0.2.2...v0.3.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-08-06 10:16:18 -06:00
carllin 1dbb5c8647
Deserialize snapshots (#5417)
* Deserialize snapshots
2019-08-05 22:53:19 -07:00
Pankaj Garg c524d62ce0
Implement coding shred generation (#5415)
* Implemenet coding shred generation

* address review comments
2019-08-05 16:32:34 -07:00
Michael Vines dd4640e1ed
Revert "Revert "Bump version to 0.18.0-pre1""
This reverts commit 42c7d57fc0.
2019-08-05 15:55:13 -07:00
Michael Vines 42c7d57fc0
Revert "Bump version to 0.18.0-pre1"
This reverts commit 14f6d5c82b.
2019-08-05 15:53:55 -07:00
Tyera Eulberg efd09ecd37 Revert fork metrics (#5427)
* Revert "Remove duplicate row (#5419)"

This reverts commit a81dd80d60.

* Revert "Log fork stake-percentage in metrics, and display (#5406)"

This reverts commit 92e419f1c7.
2019-08-05 15:53:36 -07:00
Michael Vines 14f6d5c82b
Bump version to 0.18.0-pre1 2019-08-05 15:11:44 -07:00
Tyera Eulberg 92e419f1c7
Log fork stake-percentage in metrics, and display (#5406)
* Log fork stake percentage data

* Add fork stake percentage to dashboard

* Call out parent slot
2019-08-02 19:16:23 -06:00
Pankaj Garg 80bb0158b7
Initial implementation of packet shredder (#5401)
* Initial implementation of packet shredder

* tests

* clippy

* review comments
2019-08-02 15:53:42 -07:00
Michael Vines eb3991b9ba
Replay stage log message nits (#5408) 2019-08-02 10:08:42 -07:00
Michael Vines aee63f15c2
Rename state.tgz to snapshot.tgz to match rpc service 2019-08-02 10:07:29 -07:00
dependabot-preview[bot] f03e066ec5 Bump log from 0.4.7 to 0.4.8 (#5382)
Bumps [log](https://github.com/rust-lang/log) from 0.4.7 to 0.4.8.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang-nursery/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-08-01 14:31:18 -07:00
Rob Walker f7d3f55566
fix epoch_stakes again (#5396) 2019-08-01 14:27:47 -07:00
carllin 6cb2040a1b
Snapshot Packaging Service (#5262)
* Snapshot serialization and packaging
2019-07-31 17:58:10 -07:00
sakridge c2fc0f2418
Plumb libra accounts to genesis (#5333)
* Plumb move_loader to genesis

* Remove core dependency on genesis-programs
2019-07-31 16:10:55 -07:00
Rob Walker 9278201198
fix epoch_stakes (#5355)
* fix epoch_stakes

* fix stake_state to use stakers_epoch

* don't allow withdrawal before deactivation
2019-07-31 15:13:26 -07:00
Jack May d09afdbefe
Synchronize and cleanup instruction processor lists (#5356) 2019-07-31 14:28:14 -07:00
dependabot-preview[bot] 01d2b4e952 Bump jsonrpc-http-server from 12.1.0 to 13.0.0 (#5361)
* Bump jsonrpc-http-server from 12.1.0 to 13.0.0

Bumps [jsonrpc-http-server](https://github.com/paritytech/jsonrpc) from 12.1.0 to 13.0.0.
- [Release notes](https://github.com/paritytech/jsonrpc/releases)
- [Commits](https://github.com/paritytech/jsonrpc/compare/v12.1.0...v13.0.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Update all jsonrpc crates to v13.0.0
2019-07-31 14:30:47 -06:00
Jack May 6d7cb23c61
Add command to create genesis accounts (#5343) 2019-07-30 23:43:12 -07:00
Michael Vines bd7e269280 Kill rsync (#5336)
automerge
2019-07-30 22:43:47 -07:00
Pankaj Garg b05b42d74d
Reduce max blob size (#5345)
* Reduce max blob size

* ignore test_star_network_push_rstar_200
2019-07-30 22:15:07 -07:00
dependabot-preview[bot] af733a678a Bump serde_derive from 1.0.97 to 1.0.98 (#5314)
Bumps [serde_derive](https://github.com/serde-rs/serde) from 1.0.97 to 1.0.98.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.97...v1.0.98)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-30 21:45:34 -07:00
Michael Vines 4a336eb5ff
ValidatorConfig path reform: use Path/PathBuf for paths (#5353) 2019-07-30 19:47:24 -07:00
dependabot-preview[bot] b7e08052ae Bump serde from 1.0.97 to 1.0.98 (#5315)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.97 to 1.0.98.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.97...v1.0.98)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-30 19:46:50 -07:00
Michael Vines c78db6a94b
ledger path reform: use Path/PathBuf instead of strings (#5344) 2019-07-30 15:53:41 -07:00
carllin 7dc5cc26a6
Make max_epoch check in next_leader_at in leader schedule (#5342) 2019-07-30 15:51:02 -07:00
Sagar Dhawan d7a2b790dc
Limit the size of gossip push and gossip pull response (#5348)
* Limit the size of gossip push and gossip pull response

* Remove Default::default

* Rename var
2019-07-30 15:43:17 -07:00
Pankaj Garg a7a10e12c7
Forward transactions as packets instead of blobs (#5334)
* Forward transactions as packets instead of blobs

* clippy
2019-07-30 14:50:02 -07:00
sakridge 8d243221f0
Ignore flaky local cluster tests (#5347)
* Add logging to local_cluster tests

* Ignore flaky test_leader_failure_4, test_repairman_catchup

And crashing banking benchmarks.
2019-07-30 13:48:46 -07:00
Justin Starry 84368697af
Fix metrics when leader does not report metrics (#5291) 2019-07-30 16:18:33 -04:00
sakridge a642168369
Add move to bench-tps (#5250) 2019-07-27 15:28:00 -07:00
Dan Albert b8835312bb
Update cargo.toml files to 0.18.0-pre0 (#5303) 2019-07-27 11:42:06 -06:00
Pankaj Garg aef7bae60d
Let grace ticks to roll over into multiple leader slots (#5268)
* Let grace ticks to roll over into multiple leader slots

* address review comments
2019-07-26 11:33:51 -07:00
Tyera Eulberg dab7de7496
Add confidence cache to BankForks (#5066)
* Add confidence cache to BankForks

* Include stake-weighted lockouts in cache

* Add cache test

* Move confidence cache updates to handle_votable_bank

* Prune confidence cache on prune_non_root()

* Spin thread to process aggregate_stake_lockouts

* Add long-running thread for stake_weighted_lockouts computation
2019-07-26 11:27:57 -06:00
dependabot-preview[bot] 349ebec629 Bump serde from 1.0.94 to 1.0.97 (#5285)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.94 to 1.0.97.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.94...v1.0.97)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-26 07:22:25 -06:00
Sagar Dhawan a233a1c822
Fix poh recorder not flushing virtual ticks immediately (#5277)
* Fix poh not flushing virtual ticks immediately

* Fix test_would_be_leader_soon
2019-07-25 11:08:44 -07:00
dependabot-preview[bot] 7ee8383e02 Bump serde_derive from 1.0.94 to 1.0.97 (#5279)
Bumps [serde_derive](https://github.com/serde-rs/serde) from 1.0.94 to 1.0.97.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.94...v1.0.97)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-25 06:47:24 -06:00
Sagar Dhawan 535df0026d
Fixes for Blocktree space amplification and slot deletion (#5266)
* Fixes for Blocktree space amplification and slot deletion
2019-07-24 17:28:08 -07:00
Sagar Dhawan 937816e67b Post warning if window service isn't receiving any data (#5269)
automerge
2019-07-24 12:46:10 -07:00
dependabot-preview[bot] 610a02c518 Bump jsonrpc-ws-server from 12.0.0 to 12.1.0 (#5261)
Bumps [jsonrpc-ws-server](https://github.com/paritytech/jsonrpc) from 12.0.0 to 12.1.0.
- [Release notes](https://github.com/paritytech/jsonrpc/releases)
- [Commits](https://github.com/paritytech/jsonrpc/compare/v12.0.0...v12.1.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-24 10:52:44 -06:00
Justin Starry 547a7a345f
Add logs to indicate when the leader changes (#5253) 2019-07-23 22:19:20 -04:00
Sagar Dhawan 81f4fd56c7 Log the repairee pubkey when unable to serve them (#5222)
automerge
2019-07-22 14:13:29 -07:00
dependabot-preview[bot] a6302acfd5 Bump jsonrpc-http-server from 12.0.0 to 12.1.0 (#5211)
Bumps [jsonrpc-http-server](https://github.com/paritytech/jsonrpc) from 12.0.0 to 12.1.0.
- [Release notes](https://github.com/paritytech/jsonrpc/releases)
- [Commits](https://github.com/paritytech/jsonrpc/compare/v12.0.0...v12.1.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-22 10:34:27 -06:00
Michael Vines 0d8f5379a0
Add time units 2019-07-22 08:22:21 -07:00
Sagar Dhawan a07b17b9b5
Drop older slots in the ledger (#5188)
* Add facility to delete blocktree columns in range

* Add ledger cleanup service

* Add local_cluster test
2019-07-20 13:13:55 -07:00
dependabot-preview[bot] 0958905df8 Bump reqwest from 0.9.18 to 0.9.19 (#5201)
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.9.18 to 0.9.19.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.9.18...v0.9.19)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-19 16:13:34 -06:00
Tyera Eulberg 3f54c0f1a6
Update struct order arbitrarily to match rpc output (#5197) 2019-07-19 11:45:04 -06:00
dependabot-preview[bot] 4684faa5e8 Bump jsonrpc-core from 12.0.0 to 12.1.0 (#5143)
Bumps [jsonrpc-core](https://github.com/paritytech/jsonrpc) from 12.0.0 to 12.1.0.
- [Release notes](https://github.com/paritytech/jsonrpc/releases)
- [Commits](https://github.com/paritytech/jsonrpc/compare/v12.0.0...v12.1.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-19 09:59:56 -07:00
sakridge 8b69998379
Lower recovery messages (#5181) 2019-07-19 09:20:14 -07:00
dependabot-preview[bot] 19e131d710 Bump jsonrpc-derive from 12.0.0 to 12.1.0 (#5193)
Bumps [jsonrpc-derive](https://github.com/paritytech/jsonrpc) from 12.0.0 to 12.1.0.
- [Release notes](https://github.com/paritytech/jsonrpc/releases)
- [Commits](https://github.com/paritytech/jsonrpc/compare/v12.0.0...v12.1.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-19 07:34:48 -07:00
dependabot-preview[bot] 9fd34cd985 Bump untrusted from 0.6.2 to 0.7.0 (#5194)
Bumps [untrusted](https://github.com/briansmith/untrusted) from 0.6.2 to 0.7.0.
- [Release notes](https://github.com/briansmith/untrusted/releases)
- [Commits](https://github.com/briansmith/untrusted/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-19 07:34:04 -07:00
Michael Vines adfb8ff2a1
Add getEpochInfo() and getLeaderSchedule() RPC methods (#5189)
* Add getLeaderSchedule() RPC method

* Add getEpochInfo() RPC method

* Add JSON RPC docs
2019-07-19 07:31:18 -07:00
Rob Walker afa05acb32
more replay_stage grooming (#5163) 2019-07-18 14:54:27 -07:00
Sagar Dhawan a3a91ba222 Fix misleading variable name (#5176)
automerge
2019-07-18 14:07:32 -07:00
Sagar Dhawan 751b54b60b
Skip sleeping in replay stage if a bank was recently processed (#5161)
* Skip sleeping in replay stage if a bank was recently processed

* Remove return
2019-07-18 12:04:53 -07:00
Sagar Dhawan 6ad9dc18d8
Add ability to prune ledger (#5128)
* Add utility to prune the ledger

* Add tests

* Fix clippy

* Fix off by one

* Rework to force delete every column

* Minor fixup
2019-07-17 14:42:29 -07:00
Rob Walker 027ebb6670
no more OUT_DIR (#5139)
* no more OUT_DIR

* no more OUT_DIR

* more information about failure
2019-07-17 14:27:58 -07:00
Rob Walker 0ffd91df27
groom poh_recorder (#5127)
* groom poh_recorder

* fixup

* nits

* slot() from the outside means "the slot the recorder is working on"

* remove redundant check

* review comments, put next_tick back in the "is reset" check

* remove redundant check
2019-07-17 14:10:15 -07:00
Sagar Dhawan 10d85f8366 Add weighted shuffle support for values upto u64::MAX (#5151)
automerge
2019-07-17 12:44:28 -07:00
dependabot-preview[bot] bbd0455418 Bump log from 0.4.6 to 0.4.7 (#5144)
Bumps [log](https://github.com/rust-lang/log) from 0.4.6 to 0.4.7.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang-nursery/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/compare/0.4.6...0.4.7)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-17 09:25:57 -06:00
Rob Walker f88c72c41e
stress tweaks (#5140) 2019-07-16 22:04:40 -07:00
sakridge 94db9cd412
Reduce banking_stage bench copy-paste code and fix programs bench (#4926) 2019-07-16 18:28:18 -07:00
Justin Starry bf5bce50a4
Fix stake pruning test (#5124) 2019-07-16 13:20:03 -04:00
Rob Walker fe87c05423
fix transaction_count (#5110)
* fix transaction_count

* add sig count to bank hash
2019-07-15 13:42:59 -07:00
sakridge 40faaef9da
Revert "Logging (#5017)" (#5096)
This reverts commit b50a3bae72.
2019-07-14 18:48:15 -07:00
Rob Walker 0d27515d09
tmp dirs target to farf (#5079) 2019-07-12 18:28:42 -07:00
Pankaj Garg 1c966aac25 Facility to generate a blocktree prune list using ledger tool (#5041)
automerge
2019-07-12 16:58:13 -07:00
Sagar Dhawan 7aecb87bce
Add a version field to blobs (#5057) 2019-07-12 13:43:19 -07:00
Michael Vines ebcdc06dc3
Restore ledger-tool print and json commands (#5048)
* Restore ledger-tool print and  json commands

* Remove obsolete read_ledger()
2019-07-11 20:33:36 -07:00
Sagar Dhawan 0a36a78133
Fix replicator segment selection (#5046) 2019-07-11 18:31:41 -07:00
Trent Nelson 5698d48dc8 merkle-tree: Make instantiation a little less painful (#5037)
automerge
2019-07-11 15:15:08 -07:00
Pankaj Garg 4c90898f0b
Dynamic erasure set configuration (#5018)
* Use local erasure session to create/broadcast coding blobs

* Individual session for each recovery (as the config might be different)

* address review comments

* new constructors for session and coding generator

* unit test for dynamic erasure config
2019-07-11 13:58:33 -07:00
Tyera Eulberg b2c776eabc
Fix getProgramAccounts RPC (#5024)
* Use scan_accounts to load accounts by program_id

* Add bank test

* Use get_program_accounts in RPC
2019-07-11 12:58:28 -06:00
Sagar Dhawan e4926e4110
Set exit when replicator run exits (#5016) 2019-07-10 16:27:18 -07:00
carllin b50a3bae72
Logging (#5017)
* Add logging to replay_stage

* locktower logging
2019-07-10 15:52:31 -07:00
Sagar Dhawan 35ec7a5156
Decouple turns from segments in PoRep (#5004)
* Decouple Segments from Turns in Storage

* Get replicator local cluster tests running in a reasonable amount of time

* Fix unused imports

* Document new RPC APIs

* Check for exit while polling
2019-07-10 13:33:29 -07:00
Mark E. Sinclair a383ea532f Implement new Index Column (#4827)
* Implement new Index Column

* Correct slicing of blobs

* Mark coding blobs as coding when they're recovered

* Prevent broadcast stages from mixing coding and data blobs in blocktree

* Mark recovered blobs as present in the index

* Fix indexing error in recovery

* Fix broken tests, and some bug fixes

* increase min stack size for coverage runs
2019-07-10 11:08:17 -07:00
Rob Walker f537482c86
remove set_leader from cluster_info (#4998) 2019-07-09 22:06:47 -07:00
Sagar Dhawan b8e7736af2 Move SLOTS_PER_SEGMENT to genesis (#4992)
automerge
2019-07-09 16:48:40 -07:00
Rob Walker f777a1a74c
groom replay_stage and poh_recorder (#4961)
* groom replay_stage and poh_recorder

* fixup

* fixup

* don't freeze() parent, need to review bank_forks and maybe vote...
2019-07-09 15:36:30 -07:00
carllin 22ef3c7c54
Blob verify (#4951)
* Ensure signable data is not out of range

* Add a broadcast stage that puts bad sizes in blobs

* Resign blob after modifyign size

* Remove assertions that fail when size != meta.size
2019-07-08 18:21:52 -07:00
Rob Walker 417e8d5064
fix blocktree_processor test_process_entries_stress (#4967) 2019-07-08 18:11:58 -07:00
Trent Nelson 1feb9bea21
Harden Merkle Tree against second pre-image attacks (#4925)
* merkle-tree: Harden against second pre-image attacks

* core/chacha: Bump test golden hash
2019-07-08 19:00:06 -06:00
Trent Nelson c5ba2e0883 bank_forks test stability (#4959)
automerge
2019-07-08 15:55:49 -07:00
Rob Walker eb4edd75e6
make bank commit_credits non public (#4944)
* make bank commit_credits non pub

* track down create() failure

* move bank_client to process_transaction(), which commits credits
2019-07-08 15:37:54 -07:00
Sagar Dhawan bb6bcd79c0
Handle replicator errors without panicking (#4955)
* Handle replicator errors without panicking

* Unwelcome println
2019-07-08 12:43:35 -07:00
Sagar Dhawan ef7022d638
Refactor replicators to not block on startup (#4932)
* Refactor replicators to not block on startup

* Ignore setup failure
2019-07-08 10:17:25 -07:00
carllin 2aac094f63
Ensure blobs are deserializable without unwrapping (#4948)
* Return result from deserializing blobs in blocktree instead of assuming deserialization will succeed

* Mark bad deserialization as dead fork

* Add test for corrupted blobs in blocktree and replay_stage
2019-07-07 14:37:12 -07:00
carllin fc180f4cbf
Halve stake of malicious validator (#4937) 2019-07-05 15:45:39 -07:00
dependabot-preview[bot] db13b52e6a Bump serde_json from 1.0.39 to 1.0.40 (#4881)
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.39 to 1.0.40.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.39...v1.0.40)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-03 16:15:56 -06:00
Rob Walker 2ec5d2c7f5
start local_cluster nodes from genesis blocks (#4928) 2019-07-03 14:03:52 -07:00
Sagar Dhawan 55b8ff72d0
Enable parallel tests to reduce test time (#4919)
* Add crate to serialize some tests

* Ignore unused attribute warning

* Enable parallel run in CI

* Try to fix lograte tests

* Fix interdependent counter tests
2019-07-02 17:35:03 -07:00
Rob Walker 12ef0c25b5 change vote commission to u8 (from u32) (#4887)
automerge
2019-07-02 14:18:11 -07:00
Pankaj Garg 3615445a12
Broadcast run for injecting fake blobs in turbine (#4889)
* Broadcast run for injecting fake blobs in turbine

* address review comments

* new local cluster test that uses fake blob broadcast

* added a test to make sure tvu_peers ordering is guaranteed
2019-07-01 17:54:03 -07:00
Sagar Dhawan 65adce65fa
Always send pull responses to the origin addr (#4894) 2019-07-01 16:49:05 -07:00
carllin c1953dca8f
Cleanup some of banking stage (#4878)
* Add committable transactions that cause errors like InstructionErrors back to retryable list on MaxHeightReached

* Remove unnecessary logic

* Add comments/renaming for clarity
2019-07-01 12:14:40 -07:00
Tyera Eulberg d7ed3b8024 Add RPC api to return program accounts (#4876)
automerge
2019-06-29 09:59:07 -07:00
sakridge a89589a1d5
Add Measure abstraction over measuring time intervals (#4851)
Allows one to swap in different implementations. This provides
the normal Insant::now() -> .elapsed() path.
2019-06-29 15:34:49 +02:00
carllin 75b494d4a3 Lower warn to info, fetch from validator root instead of root + 1 (#4870)
* Lower warn to info, fetch from validator root instead of root + 1

* b/c -> because
2019-06-28 16:17:20 -07:00
dependabot-preview[bot] f0191a98ab Bump serde from 1.0.93 to 1.0.94 (#4864)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.93 to 1.0.94.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.93...v1.0.94)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-06-28 15:07:16 -07:00
sakridge 0fa1af5d47
Cleanup num_threads() and batch_limit numbers (#4852) 2019-06-28 10:55:24 +02:00
Tyera Eulberg 66552d7047
Credit-Only Accounts: Cache account balance for thread-safe load/store (#4691)
* Implement CreditOnlyLocks

* Update credit-only atomic on account load

* Update credit-only atomic after bank.freeze_lock; store credits if all credit-only lock references are dropped

* Commit credit-only credits on bank freeze

* Update core to CreditAccountLocks

* Impl credit-only in System Transfer

* Rework CreditAccountLocks, test, and fix bugs

* Review comments: Pass CreditAccountLocks by reference; Tighten up insert block

* Only store credits on completed slot

* Check balance in bench_exchange funding to ensure commit_credits has completed

* Add is_debitable info to KeyedAccount meta to pass into programs

* Reinstate CreditOnlyLocks check on lock_account

* Rework CreditAccountLocks to remove strong_count usage

* Add multi-threaded credit-only locks test

* Improve RwLocks usage

* Review comments: panic if bad things happen; tighter code

* Assert lock_accounts race does not happen

* Revert panic if bad things happen; not a bad thing
2019-06-27 17:25:10 -04:00
dependabot-preview[bot] 979df17328 Bump serde_derive from 1.0.93 to 1.0.94 (#4856)
Bumps [serde_derive](https://github.com/serde-rs/serde) from 1.0.93 to 1.0.94.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.93...v1.0.94)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-06-27 13:18:19 -07:00
carllin 6cec61dcfc
Re-enable tests (#4848) 2019-06-27 12:09:14 -07:00
sakridge a9b044f0ab
Fix banking_stage benchmark sends (#4850)
Only one big batch was being sent so only 1
thread active at a time in the benchmark.
2019-06-27 10:37:33 +02:00
sakridge fbea9d8621
Page-pin packet memory for cuda (#4250)
* Page-pin packet memory for cuda

Bring back recyclers and pin offset buffers

* Add packet recycler to streamer

* Add set_pinnable to sigverify vecs to pin them

* Add packets reset test

* Add test for recycler and reduce the gc lock critical section
* Add comments/tests to cuda_runtime

* Add recycler to recv_blobs path.

* Add trace/names for debug and PacketsRecycler to bench-streamer

* Predict realloc and unpin beforehand.

* Add helper to reserve and pin

* Cap buffered packets length

* Call cuda wrapper functions
2019-06-27 09:32:32 +02:00
carllin 97c97db97e
Fix early exit clearing all buffered packets (#4810) 2019-06-26 22:39:50 -07:00
carllin 9a52b01171
Change to crossbeam channel in banking_threads VerifiedReceiver (#4822)
* Add crossbeam channel instead of channel in banking_stage
2019-06-26 18:42:27 -07:00
Sagar Dhawan df1c473341
Add storage point tracking and tie in storage rewards to economics (#4824)
* Add storage point tracking and tie in storage rewards to epochs and economics

* Prevent validators from updating their validations for a segment

* Fix test

* Retain syscall scoping for readability

* Update Credits to own epoch tracking
2019-06-26 10:40:03 -07:00
Rob Walker 8a64e1ddc3
add fee burning (#4818) 2019-06-26 10:13:21 -07:00
dependabot-preview[bot] eb47538a82 Bump chrono from 0.4.6 to 0.4.7 (#4812)
Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.6 to 0.4.7.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/master/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.6...v0.4.7)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-06-26 07:58:41 -06:00
Justin Starry 861d6468ca Stake weighted pruning for the gossip network (#4769)
* Stake weighted pruning

* Fix compile error

* Fix clippy errors

* Add helper for creating a connected staked network

* Bug fixes and test groundwork

* Small refactor

* Anatoly's feedback and tests

* Doc updates

* @rob-solana's feedback

* Fix test bug and add log trace

* @rob-solana's feedback
2019-06-26 00:30:16 -07:00
carllin d6737b8cc9
Set epoch schedule in set_root in leader schedule cache (#4821) 2019-06-26 00:19:48 -07:00
Trent Nelson f20ba423ca
Merklize PoH TX mixin hash (#4644) 2019-06-25 14:44:27 -06:00
Sathish 9e7f618cff Set proper count value for account stores (#4797)
* set count values for store accounts

* Use AppendVecId type
2019-06-25 07:21:45 -07:00
Sagar Dhawan 74a06e4230
Update thinclient to resend the same tx until its blockhash expires (#4807) 2019-06-24 16:46:34 -07:00
Pankaj Garg 3f8ff23125
Forward transactions to the leader for next Nth slot (#4806)
* review comments
2019-06-24 15:56:50 -07:00
anatoly yakovenko 29611fb61d
tower consensus naming (#4598)
s/locktower/tower/g
2019-06-24 13:41:23 -07:00
Michael Vines 206e62271b Ignore flaky test_two_unbalanced_stakes (#4794)
automerge
2019-06-23 20:55:43 -07:00
dependabot-preview[bot] 4e78354ab6 Bump serde_derive from 1.0.92 to 1.0.93 (#4790)
Bumps [serde_derive](https://github.com/serde-rs/serde) from 1.0.92 to 1.0.93.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.92...v1.0.93)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-06-23 20:23:41 -07:00
dependabot-preview[bot] 0e1480b84e Bump serde from 1.0.92 to 1.0.93 (#4791)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.92 to 1.0.93.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.92...v1.0.93)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-06-23 17:23:41 -07:00
Sagar Dhawan 11992946a4
Add storage reward pools (#4779) 2019-06-22 17:18:35 -07:00
Dan Albert 9911942dbd Increment cargo.toml files to v0.17.0 2019-06-22 04:35:25 +00:00
Trent Nelson 8acbb4ab2f Bank cap rpc (#4774)
* core/rpc: Name magic number for minted lamports in tests genesis block

* core/rpc: Expose bank::capitalization() via getSolTotalSupply RPC method

* book: Add entry for getTotalSupply RPC method
2019-06-21 21:00:26 -07:00
Rob Walker a49f5378e2
rewrite vote credits redemption to eat from rewards_pools on an epoch-sensitive basis (#4775)
* move redemption to rewards pools

* rewrite redemption, touch a few other things

* re-establish test coverage
2019-06-21 20:43:24 -07:00
Michael Vines f39e74f0d7 serde the full FeeCalculator (#4778)
automerge
2019-06-21 17:23:26 -07:00
Sagar Dhawan 22b767308a
Add insturctions to run a replicator on testnet (#4733) 2019-06-21 16:32:23 -07:00
Michael Vines 36aa876833 Avoid linking with CUDA directly 2019-06-21 15:26:22 -07:00
carllin 06ba0b7279
Remove holding cluster_info lock while forwarding packets (#4773) 2019-06-21 15:21:49 -07:00
Sagar Dhawan da925142d1 Update replicator ports and silence socket timeout on windows (#4770)
automerge
2019-06-21 11:28:52 -07:00
dependabot-preview[bot] deb83cdef6 Bump rayon from 1.0.3 to 1.1.0 (#4729)
automerge
2019-06-21 09:32:41 -07:00
dependabot-preview[bot] 20db335aed Bump reqwest from 0.9.17 to 0.9.18
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.9.17 to 0.9.18.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.9.17...v0.9.18)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-06-21 08:50:49 -07:00
Michael Vines 36c9e22e3d Revert "Dynamic erasure (#4653)"
This reverts commit ada4d16c4c.
2019-06-20 20:53:03 -07:00
Mark E. Sinclair ada4d16c4c
Dynamic erasure (#4653)
Remove erasure-related constants

Remove unneeded `Iterator::collect` call

Document erasure module

Randomize coding blobs used for repair
2019-06-20 20:27:41 -05:00
carllin aacb38864c
Mark dead forks in replay stage (#4715)
* Add DeadSlots column family

* Filter dead forks from get_slots_since

* Mark erroring slots as dead in replay stage, add test

* Mark dead forks in progress instead of removing them

* Fix logging process_entries failures in replay_stage

* Unignore test_fail_entry_verification_leader
2019-06-20 15:50:41 -07:00
Michael Vines 4a32bc48d2 ignore unstable test_repairman_catchup 2019-06-20 09:24:11 -07:00
Pankaj Garg 9800e09431
Thread pool for par_iter in EntrySlice::verify (#4732)
* Use thread pool for entry verify par iter

* some performance metrics

* check batch size and use CPU for smaller batches
2019-06-19 16:31:32 -07:00
carllin 61483c18ca
Change seed for retransmit to use blob signature (#4727)
* Switch seed for retransmit to use blob signature

* Use seed_len

* Use last bytes of signature as seed instead of first bytes
2019-06-19 15:36:06 -07:00
Rob Walker 41fbdc6e08
use stake (#4721) 2019-06-19 11:54:52 -07:00
carllin 46bb79df29
Support for custom BroadcastStage in local cluster tests (#4716)
* Refactor BroadcastStage to support custom implementations, add FailEntryVerificationBroadcastRun implementation

* Plumb switch on broadcast type through validator

* Add test for validator generating non-verifiable entries to local_cluster

* Fix bad initializers

* Refactor broadcast run code into utils
2019-06-19 00:13:19 -07:00
dependabot-preview[bot] 2da7c7fbd3 Bump nix from 0.14.0 to 0.14.1 (#4642)
automerge
2019-06-18 11:36:26 -07:00
Michael Vines 5683282c94 Update to solana-perf-libs v0.14.0, with support for both CUDA 10.0 and 10.1 2019-06-18 10:41:03 -07:00
Sathish Ambley 8b41a5d725 periodically save config in separate folders 2019-06-17 22:48:27 -07:00
Sathish Ambley 7fd879b417 Restart validator nodes from snapshots 2019-06-17 22:48:27 -07:00
Sathish Ambley dc5c6e7cf8 validator restart 2019-06-17 22:48:27 -07:00
Michael Vines feeaad619a Avoid panic if no rpc peers exist 2019-06-17 19:47:45 -07:00
sakridge 9cafd1f85e
Change check_txs to ignore recv errors and re-enable test (#4593)
Use more chunks to avoid duplicate signature failures:
Duplicate signatures can occur because bank.clear_signatures()
can occur before the bank has actually committed the signatures
to the status cache and then error out on the next set of transactions.
2019-06-17 19:04:21 -07:00
Sagar Dhawan cc48773b03 Add "download from replicator" utility (#4709)
automerge
2019-06-17 18:12:13 -07:00
Sagar Dhawan 8fbf0e2d9f
Update replicators to use the storage blockhash to generate offsets (#4712) 2019-06-17 16:39:26 -07:00
Michael Vines 5bf87de136 Add obvious log message indicating CUDA feature state 2019-06-17 11:01:55 -07:00
Michael Vines 97a136ea20 Set rustc-cfg=cuda explicitly, also code cleanup 2019-06-17 11:01:55 -07:00
Rob Walker 4c6a6d63bf
add MiningPools, fund validator MiningPools from inflation (#4676)
* add MiningPool fund validator MinigPools from inflation

* fixup

* finish rename of MINIMUM_SLOT_LENGTH to MINIMUM_SLOTS_PER_EPOCH

* deterministic miningpool location

* point_value, not credit_value... use f64
2019-06-14 11:38:37 -07:00
Sagar Dhawan 119467df59
Add storage mining pool to genesis and implement automatic reward redeeming (#4683)
* Add storage mining pool to genesis and implement automatic reward collection

* Address review comments
2019-06-13 22:30:51 -07:00
Sagar Dhawan b1ac8f933b
Fix storage program space issues and limit storage transaction data (#4677) 2019-06-13 17:53:54 -07:00
Michael Vines 8628f33d0b Fix HostId field in the testnet dashboard 2019-06-13 16:09:09 -07:00
carllin 8c1b9a0b67
Data plane verification (#4639)
* Add signature to blob

* Change Signable trait to support returning references to signable data

* Add signing to broadcast

* Verify signatures in window_service

* Add testing for signatures to erasure

* Add RPC for getting current slot, consume RPC call in test_repairman_catchup for more deterministic results
2019-06-12 16:43:05 -07:00
Michael Vines d807217be7
Simplify and camelCase getEpochVoteAccounts RPC API (#4658)
* Simplify and camelCase getEpochVoteAccounts RPC API

* Set a commission for testing
2019-06-12 14:12:08 -07:00
Sagar Dhawan 8019bff391 Fixes for storage program and rework storage stage (#4654)
automerge
2019-06-11 18:27:47 -07:00
Rob Walker 697228a484
rpc vote_accounts by ecurrent pocch, not stakers epoch (#4651) 2019-06-11 16:57:47 -07:00
Rob Walker 3217a1d70c
use highest staked node as bootstrap leader, remove bootstrap_leader from genesis_block (#4635)
* use highest staked node as bootstrap leader, remove bootstrap_leader from genesis_block

* clippy

* fixup

* fixup
2019-06-11 11:44:58 -07:00
dependabot-preview[bot] 1d957b6b80 Bump byteorder from 1.3.1 to 1.3.2 (#4641)
Bumps [byteorder](https://github.com/BurntSushi/byteorder) from 1.3.1 to 1.3.2.
- [Release notes](https://github.com/BurntSushi/byteorder/releases)
- [Changelog](https://github.com/BurntSushi/byteorder/blob/master/CHANGELOG.md)
- [Commits](https://github.com/BurntSushi/byteorder/compare/1.3.1...1.3.2)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-06-11 11:21:26 -07:00
Rob Walker e56430c9fb
make runtime depend on bpf_loader (#4601)
* make runtime depend on bpf_loader

* remove vote redundancy, move bpf_loader to genesis, export program\! from bpf_loader crate

* move bpf_loader specification into genesis

* bpf tests to use genesis with bpf

* need to avoid depending on programs, except for macros
2019-06-11 10:27:22 -07:00
Michael Vines e4d8ea11ac
Make lamports_per_signature dynamic based on cluster load (#4562)
* Make lamports_per_signature dynamic based on cluster load

* Move transaction-fees.md to implemented
2019-06-10 22:18:32 -07:00
carllin a4035a3c65
Remove record locks and parent locks from accounts (#4633)
* Revert "Fix parent record locks usage in child banks (#4159)"

This reverts commit 69eeb7cf08.

* Revert "Fix DuplicateSignatures caused by races on frozen banks (#3819)"

This reverts commit 083090817a.

* Remove unused imports
2019-06-10 22:05:46 -07:00
dependabot-preview[bot] 64d63966c7 Bump jsonrpc crates to 12.0.0 (#4553)
* Bump jsonrpc-pubsub from 11.0.0 to 12.0.0

Bumps [jsonrpc-pubsub](https://github.com/paritytech/jsonrpc) from 11.0.0 to 12.0.0.
- [Release notes](https://github.com/paritytech/jsonrpc/releases)
- [Commits](https://github.com/paritytech/jsonrpc/commits/v12.0.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Update all jsonrpc crates
2019-06-10 13:22:57 -06:00
Rob Walker a18c0e34f4
add activate_stake to stake_api (#4600) 2019-06-10 12:17:29 -07:00
Michael Vines 0dbe5ee559
Add chacha-sys crate (#4620)
* af9ff9c7f9/src/cpu-crypt

* Add chacha-sys crate

* Remove chacha feature

* Remove erasure feature

* Add .gitignore
2019-06-10 07:14:02 -07:00
TristanDebrunner 0c4cb76acf
Add GPU based PoH verification (#4524)
* Add GPU poh verify

* Switch to single PoH verify function

* Add EntrySlice verify tests with hashes and txs

* Add poh-verify benchmarks
2019-06-08 10:21:43 -06:00
Pankaj Garg ac6dbf8f04
Broadcast blobs even if the peers have no stake (#4597) 2019-06-07 14:12:27 -07:00
carllin c9d63204eb
Replace unneeded seqcst with relaxed on atomic operations (#4587) 2019-06-06 23:53:21 -07:00
Michael Vines cfab54511b Ignore bench_banking_stage_multi_accounts (#4590)
automerge
2019-06-06 21:47:07 -07:00
Pankaj Garg 194491ae96
Removed some dead code (#4563)
* Removed some dead code

* remove dead code from Replicator
2019-06-06 14:26:12 -07:00
anatoly yakovenko 82df267ec9
s/avalanche/turbine (#4561)
* s/avalanche/turbine/g
2019-06-06 12:48:40 -07:00
Michael Vines 51aacfe3ca
Use pure-rust reed-solomon-erasure for windows (#4548) 2019-06-04 21:49:27 -07:00
Michael Vines 8ec5a47027
Add EntryWriter::write() stub for windows (#4546) 2019-06-04 20:15:37 -07:00
carllin 5bd3eb4557
Up number of threads (#4541) 2019-06-04 18:01:28 -07:00
dependabot-preview[bot] e9cb4a12dc Bump serde_derive from 1.0.91 to 1.0.92 (#4505)
automerge
2019-06-04 15:48:23 -07:00
Sagar Dhawan de5cad9211
Add account owner to Storage Accounts (#4537)
* Add account owner to Storage Accounts

* Fix tests
2019-06-04 14:52:52 -07:00
dependabot-preview[bot] ce2ce76958 Bump serde from 1.0.91 to 1.0.92 (#4504)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.91 to 1.0.92.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.91...v1.0.92)
2019-06-04 14:32:09 -07:00
dependabot-preview[bot] d77c98530f Bump libc from 0.2.55 to 0.2.58 (#4514)
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.55 to 0.2.58.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.55...0.2.58)
2019-06-04 14:31:06 -07:00
Michael Vines 3635a68129 solana-install now compiles for Windows (#4531)
automerge
2019-06-04 08:51:20 -07:00
Pankaj Garg 70a16e91a5
Randomize avalanche broadcast peer table for each blob (#4529)
* fix clippy warnings
2019-06-03 20:38:05 -07:00
Sagar Dhawan a77775cb58 Move validation submissions into its own fn (#4528)
automerge
2019-06-03 18:27:06 -07:00
Sagar Dhawan 167e15a5ae
Update replicator sampling and proof generation (#4522)
* Update replicator sampling and proof generation

* Clippy
2019-06-03 17:27:28 -07:00
Rob Walker c56052ff16
remove from_account from stake_instruction (#4502) 2019-06-03 09:04:51 -07:00
Pankaj Garg 3574469052
Add random distribution for avalanche peers (#4493)
* Add random distribution for avalanche peers

* fix clippy warnings

* bug fixes

* nits
2019-06-01 07:55:43 -07:00
Rob Walker 028e111fbc
remove payer from vote instructions (#4475) 2019-05-31 11:45:17 -07:00
Sathish 182096dc1a
Create bank snapshots (#4244)
* Revert "Revert "Create bank snapshots (#3671)" (#4243)"

This reverts commit 81fa69d347.

* keep saved and unsaved copies of status cache

* fix format check

* bench for status cache serialize

* misc cleanup

* remove appendvec storage on purge

* fix accounts restore

* cleanup

* Pass snapshot path as args

* Fix clippy
2019-05-30 21:31:35 -07:00
Sagar Dhawan b8aff218e2 Shutdown all services before bailing replicator init (#4487)
automerge
2019-05-30 14:36:47 -07:00
carllin 045d4d5294
Unignore test test_repairman_catchup (#4484) 2019-05-30 13:21:12 -07:00
Michael Vines 39b40dfff8
Remove runtime dependency on storage (#4480) 2019-05-30 10:54:28 -07:00
carllin c82f4a1b6d
Unignore test_repairman_catchup 2019-05-29 21:59:41 -07:00
Michael Vines 7a021dff05
Beautify Cargo.tomls with |cargo tomlfmt| (#4477) 2019-05-29 18:30:49 -07:00
carllin 348c2263ba
Remove genesis blockhash (#4471)
* Remove genesis blockhash

* Remove genesis blockhash from tests

* Fix golden
2019-05-29 17:29:02 -07:00
Pankaj Garg b5324063f1
Use thread pools for rayon par_iter (#4473)
* Use thread pools for rayon par_iter

* address review comments

* cleanup
2019-05-29 17:16:36 -07:00
Sagar Dhawan 6ed071c4dd
Fix storage stage operating on empty slots (#4474)
* Fix storage stage operating on empty slots

* Reduce fn argument count

* Fix tests
2019-05-29 15:01:20 -07:00
sakridge 4404634b14
Coalesce packets better (#4456) 2019-05-29 12:17:50 -07:00
Rob Walker 6a1de33138
tighten up packets_to_blobs (#4464)
* tighten up packets_to_blobs

* missed a test
2019-05-29 10:08:35 -07:00
carllin 534244b322
Fix set_roots to use cached bank parents instead of searching blocktree (#4466) 2019-05-29 09:43:22 -07:00
Sagar Dhawan 335dfdc4d5
Fix Gossip skipping push for some values (#4463)
* Make gossip skip over values from Pruned nodes

* Add test and init blooms to contain the origin
2019-05-28 18:39:40 -07:00
Pankaj Garg c8a03c7b3d
Save RNG for generating random storage sampling offsets (#4450)
* Save RNG for generating random storage sampling offsets

* fix clippy

* fix stable-perf

* fix chacha
2019-05-28 14:14:46 -07:00
sakridge 5340800cea
Add some optimizing to ThinClient (#4112)
Can create a multi-socketed ThinClient which will use request time
from get_recent_blockhash to tune for the best node to talk to.
2019-05-27 20:54:44 -07:00
dependabot-preview[bot] 13c2e50b38 Bump sys-info from 0.5.6 to 0.5.7 (#4445)
automerge
2019-05-27 20:31:12 -07:00
carllin 65f89d6729
Bump logging level of validator procsesing errors (#4442) 2019-05-27 16:19:38 -07:00
Michael Vines ed6a438c51
v0.16.0 2019-05-26 19:42:15 -07:00
Michael Vines 2adb98a4a0 Ignore flaky test_repairman_catchup (#4439)
automerge
2019-05-26 12:24:20 -07:00
carllin aa3c00231a Fix should_update check to update EpochSlots in gossip (#4435)
automerge
2019-05-25 06:44:40 -07:00
carllin d772a27936 Plumb ClusterInfoRepairListener (#4428)
automerge
2019-05-24 19:20:09 -07:00
Rob Walker 0302f13b97
add datapoint for corrupt vote_account (#4424) 2019-05-24 18:34:56 -07:00
Pankaj Garg 1f71d05299 remove copying of forwarded packets (#4425)
automerge
2019-05-24 17:35:09 -07:00
Sagar Dhawan 4c9b7c9d2b
Submit all incoming proofs as valid (#4377) 2019-05-24 14:49:10 -07:00
carllin cf4bb70d80 Rename id to pubkey in cluster_info_repair_listener (#4421) 2019-05-24 04:31:32 -07:00
carllin 57f8a15b96
Fix issues in ClusterInfoRepairListener (#4418)
* Sort repairmen before shuffling so order is the same across all validators

* Reduce repair redundancy to 1 for now

* Fix local cache of roots so that 1) Timestamps are only updated to acknowledge a repair was sent 2) Roots are updated even when timestamps aren't updated to keep in sync with network

* Refactor code, add test
2019-05-24 00:47:51 -07:00
Michael Vines cfe5afd34c
_id => _pubkey variable renaming (#4419)
* wallet: rename *_account_id to *_account_pubkey

* s/from_id/from_pubkey/g

* s/node_id/node_pubkey/g

* s/stake_id/stake_pubkey/g

* s/voter_id/voter_pubkey/g

* s/vote_id/vote_pubkey/g

* s/delegate_id/delegate_pubkey/g

* s/account_id/account_pubkey/g

* s/to_id/to_pubkey/g

* s/my_id/my_pubkey/g

* cargo fmt

* s/staker_id/staker_pubkey/g

* s/mining_pool_id/mining_pool_pubkey/g

* s/leader_id/leader_pubkey/g

* cargo fmt

* s/funding_id/funding_pubkey/g
2019-05-23 23:20:04 -07:00
Michael Vines 94beb4b8c2
More fullnode -> validator renaming (#4414)
* s/fullnode_config/validator_config/g

* s/FullnodeConfig/ValidatorConfig/g

* mv core/lib/fullnode.rs core/lib/validator.rs

* s/Fullnode/Validator/g

* Add replicator-x.sh

* Rename fullnode.md to validator.md

* cargo fmt
2019-05-23 22:05:16 -07:00
Rob Walker 35e8f966e3
add freeze_lock() and fix par_process_entries() failure to detect self conflict (#4415)
* add freeze_lock and fix par_process_entries failure to detect self conflict

* fixup

* fixup
2019-05-23 17:35:15 -07:00
Michael Vines b37d2fde3d
Add storage mining pool (#4364)
* Add storage mining pool

* Set gossip port

* Add create-storage-mining-pool-account wallet command

* Add claim-storage-reward wallet command

* Create storage account upfront

* Add storage program to genesis

* Use STORAGE_ACCOUNT_SPACE

* Fix tests

* Add wallet commands to create validator/replicator storage accounts

* Add create_validator_storage_account()

* Storage stage no longer implicitly creates a storage account
2019-05-23 14:50:23 -07:00
carllin 6a9e0bc593 Change EpochSlots to use BtreeSet so that serialization/deserialization returns the same order (#4404)
automerge
2019-05-23 03:50:41 -07:00
carllin 591fd72e0b
Implement listener for serving repairs through Repairman protocol (#4306)
* Make listener for serving repairs through Repairman protocol
2019-05-23 03:10:16 -07:00
Michael Vines 2ed77b040a
create_genesis_block() now returns a struct (#4403) 2019-05-22 20:39:00 -07:00
Pankaj Garg b8f6c17dee Don't filter transactions if we are buffering it locally (#4395)
automerge
2019-05-22 17:54:28 -07:00
Pankaj Garg 36019cb1e3
Tweaks to real PoH based on perf testing (#4396)
* Some counters for real poh perf analysis

* more metrics

* Comment on CPU affinity change, and reduce hash batch size based on TPS perf

* review comments
2019-05-22 15:54:24 -07:00
Tyera Eulberg 99d2428041
Transaction format changes toward Credit-Only accounts (#4386)
* Add num_readonly_accounts slice

* Impl programs in account_keys

* Emulate current account-loading functionality using program-account_keys (breaks exchange_program_api tests)

* Fix test

* Add temporary exchange faucet id

* Update chacha golden

* Split num_credit_only_accounts into separate fields

* Improve readability

* Move message field constants into Message

* Add MessageHeader struct and fixup comments
2019-05-22 18:23:16 -04:00
Pankaj Garg eef2bdf690 Add CPU affinity for PoH service thread (#4394)
automerge
2019-05-22 14:21:43 -07:00
dependabot[bot] 190656967d Bump nix from 0.13.0 to 0.14.0 (#4382)
Bumps [nix](https://github.com/nix-rust/nix) from 0.13.0 to 0.14.0.
- [Release notes](https://github.com/nix-rust/nix/releases)
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nix-rust/nix/compare/v0.13.0...v0.14.0)

Signed-off-by: dependabot[bot] <support@dependabot.com>
2019-05-22 14:07:35 -07:00
sakridge 1d7a758c97
Bump perf libs version to fix out buffer sizing (#4385) 2019-05-22 13:00:03 -07:00
Rob Walker 578c2ad3ea
add bank hash to votes (#4381) 2019-05-21 21:45:38 -07:00
sakridge 52d453d06f
Reduce broadcast prints (#4368) 2019-05-21 17:02:19 -07:00
Sagar Dhawan c9ba9e4eb7 Add storage space constant (#4366)
automerge
2019-05-21 11:07:13 -07:00