Commit Graph

61 Commits

Author SHA1 Message Date
Sathish Ambley 4149f7fd1c Fix review comments 2019-02-27 08:22:52 -08:00
Stephen Akridge 180d297df8 Rebase and panic with no accounts
Add Accounts::has_accounts function for hash_internal_state calculation.
2019-02-27 08:22:52 -08:00
Pankaj Garg 789fff2ae2 Replace LeaderScheduler with LeaderScheduler1 (#2962)
* Migrate to LeaderScheduler1 (and added some missing methods)
* Delete LeaderScheduler
* Rename LeaderScheduler1 to LeaderScheduler
2019-02-26 22:16:18 -07:00
Sagar Dhawan a484c87354
Make gossip selection stake based (#2848) 2019-02-20 17:08:56 -08:00
carllin 1278396bd5
Cleanup consecutive entries code from window_service (#2697)
* Remove returning entries from db_ledger on insert

* Fix tests to check for correctness

* Delete generate_repairs and max_repair_entry_height
2019-02-08 14:19:28 -08:00
carllin 6e7c5f205b
Rename db_ledger to blocktree (#2698) 2019-02-07 20:52:39 -08:00
carllin fd7db7a954
Support multiple forks in the ledger (#2277)
* Modify db_ledger to support per_slot metadata, add signal for updates, and add chaining to slots in db_ledger

* Modify replay stage to ask db_ledger for updates based on slots

* Add repair send/receive metrics

* Add repair service, remove old repair code

* Fix tmp_copy_ledger and setup for tests to account for multiple slots and tick limits within slots
2019-02-07 15:10:54 -08:00
Michael Vines c5a74ada05 leader_scheduler: remove bootstrap_height 2019-02-06 14:23:10 -08:00
carllin 3feda8a315
ReplayStage asking ledger for updates (#2597)
* Modify replay stage to ask db_ledger for updates instead of reading from upstream channel

* Add signal for db_ledger to update listeners about updates

* fix flaky test
2019-02-04 15:33:43 -08:00
Greg Fitzgerald dad0bfe447 Replace transaction traits with structs
Also:
* SystemTransaction::new -> new_account
* SystemTransaction::new_create -> new_program_account
2019-02-01 11:38:25 -08:00
Michael Vines 9767468b7f Remove unneeded Option 2019-01-31 13:53:59 -08:00
Pankaj Garg 32162ef0f1
Connect TPU's broadcast service with TVU's blob fetch stage (#2587)
* Connect TPU's broadcast service with TVU's blob fetch stage

- This is needed since ledger is being written only in TVU now

* fix clippy warnings

* fix failing test

* fix broken tests

* fixed failing tests
2019-01-31 13:43:22 -08:00
Michael Vines bfaf5634a1 .unwrap() in tests instead of assert!()ing .is_ok() for a better failure message 2019-01-28 16:10:32 -08:00
Michael Vines 122a5b2f69
dedup the 2019-01-22 09:47:43 -08:00
Mark f37eb533f1
Replicator timeout (#2480)
* Add timeout to Replicator::new; used when polling for leader

* Add timeout functionality to replicator ledger download

Shares the same timeout as polling for leader

Defaults to 30 seconds

* Add docs for Replicator::new
2019-01-21 15:37:41 -06:00
Stephen Akridge 1fd7bd7ede Storage fixes
* replicators generate their sample values
* fixes to replicator block height logic
2019-01-18 13:05:35 -08:00
Tyera Eulberg 8af61f561b
Improve Wallet coverage (#2385)
* Add trait for RpcRequestHandler trait for RpcClient and add MockRpcClient for unit tests

* Add request_airdrop integration test

* Add timestamp_tx, witness_tx, and cancel_tx to wallet integration tests; add wallet integration tests to test-stable

* Add test cases

* Ignore plentiful sleeps in unit tests
2019-01-14 00:10:03 -07:00
Stephen Akridge 0c90e1eff6 Make entry_sender optional on window_service
window_service in replicator has no need to consume the the produced entries.
2019-01-09 15:15:47 -08:00
Stephen Akridge 45b4cf2887 Remove store_ledger_stage which is no longer needed 2019-01-09 15:15:47 -08:00
Stephen Akridge 4b3d64ec9f Convert chacha_encrypt_file to work with db_ledger blobs directly 2019-01-08 16:00:39 -08:00
Michael Vines 58f2598d5d Revert "Validators make a transaction to advertise their storage last_id"
This reverts commit a1759aed19.
2018-12-23 14:02:09 -08:00
Stephen Akridge a1759aed19 Validators make a transaction to advertise their storage last_id
* Also implement more storage contract logic
* Add transactions for proof validation,
* Move storage state members into system storage account userdata
2018-12-21 15:45:30 -08:00
Michael Vines 37d7ad819b Purge DB::destroy() usage 2018-12-20 10:38:03 -08:00
Rob Walker a65022aed7
DbLedger doesn't need to be mut, doesn't need an RwLock (#2215)
* DbLedger doesn't need to be mut, doesn't need an RwLock

* fix erasure cases
2018-12-18 15:18:57 -08:00
Michael Vines 6ac466c0a4 Move src/logger.rs into logger/ crate to unify logging across the workspace 2018-12-14 13:10:43 -08:00
Stephen Akridge 7cdbbfa88e Storage stage updates
* Remove logging init from storage program: saw a crash in a test
  indicating the logger being init'ed twice.
* Add entry_height mining proof to indicate which segment the result is
  for
* Add an interface to get storage miner pubkeys for a given entry_height
* Add an interface to get the current storage mining entry_height
* Set the tvu socket to 0.0.0.0:0 in replicator to stop getting entries
  after the desired ledger segment is downloaded.
* Use signature of PoH height to determine which block to download for
  replicator.
2018-12-13 11:30:12 -08:00
Carl a05a378db4 cleanup 2018-12-12 19:12:51 -08:00
Carl 245362db96 Make a dummy version of serving repairs from db_ledger 2018-12-12 19:12:51 -08:00
Greg Fitzgerald 5e703dc70a Free up the term 'replicate' for exclusive use in replicator
Also, align Sockets field names with ContactInfo.
2018-12-10 15:26:43 -07:00
Stephen Akridge 172e511e56 Use retry_transfer to test multiple times for replicator tokens
May fix failures in CI where replicator is trying to do an airdrop.
2018-12-10 12:19:00 -08:00
Greg Fitzgerald 0ef099421c cargo fmt 2018-12-08 23:19:55 -07:00
Greg Fitzgerald c49e2f8bbd cargo +nightly fix --features=bpf_c,cuda,erasure,chacha --edition 2018-12-08 23:19:55 -07:00
Greg Fitzgerald ec5a8141eb cargo fix --edition 2018-12-08 23:19:55 -07:00
Greg Fitzgerald 0a83b17cdd
Upgrade to Rust 1.31.0 (#2052)
* Upgrade to Rust 1.31.0
* Upgrade nightly
* Fix all clippy warnings
* Revert relaxed version check and update
2018-12-07 20:01:28 -07:00
Stephen Akridge 3441d3399b Replicator rework
* Move more of the replicator logic into the replicator class
* Add support for the RPC interface to query the storage last_id value
  that the replicator would sign and use to pick a block.
* Fix replicator connecting to gossip and change test to exercise that
  scenario.
2018-12-07 15:20:36 -08:00
Greg Fitzgerald 97b1156a7a Rename Ncp to GossipService
And BroadcastStage to BroadcastService since it's not included in the
TPU pipeline.
2018-12-06 15:48:19 -07:00
Michael Vines c99f93e40a Remove signature.rs indirection 2018-12-03 13:50:00 -08:00
Stephen Akridge 978fd6858f Move replicator_startup_test to integration test set
Sometimes fails when run multithreaded with other tests.
2018-11-29 11:39:42 -08:00
carllin 57a384d6a0
Rocks db window service (#1888)
* Add db_window module for windowing functions from RocksDb

* Replace window with db_window functions in window_service

* Fix tests

* Make note of change in db_window

* Create RocksDb ledger in bin/fullnode

* Make db_ledger functions generic

* Add db_ledger to bin/replicator
2018-11-24 19:32:33 -08:00
Rob Walker 3d113611cc
remove Result<> return from ClusterInfo::new() (#1869)
strip Result<> for ClusterInfo::new()
2018-11-19 11:25:14 -08:00
Michael Vines d96a6b42a5 Move drone into its own crate 2018-11-16 20:42:21 -08:00
anatoly yakovenko a41254e18c
Add scalable gossip library (#1546)
* Cluster Replicated Data Store

Separate the data storage and merge strategy from the network IO boundary.
Implement an eager push overlay for transporting recent messages.

Simulation shows fast convergence with 20k nodes.
2018-11-15 13:23:26 -08:00
Tyera Eulberg f683817b48 Remove RPU; replace with RPC 2018-11-05 20:30:47 -07:00
carllin 298bd6479a
Add first leader to genesis (#1681)
* Add first leader to genesis entries, consume in genesis.sh

* Set bootstrap leader in the bank on startup, remove instantiation of bootstrap leader from bin/fullnode

* Remove need to initialize bootstrap leader in leader_scheduler, now can be read from genesis entries

* Add separate interface new_with_leader() in mint for creating genesis leader entries
2018-11-02 14:32:05 -07:00
Rob Walker 13bfdde228
remove ledger tail code, WINDOW_SIZE begone (#1617)
* remove WINDOW_SIZE, use window.window_size()
* move ledger tail, redundant with ledger-based repair
2018-10-30 10:05:18 -07:00
carllin f6c8e1a4bf
Vote contract (#1552)
* Add Vote Contract

* Move ownership of LeaderScheduler from Fullnode to the bank

* Modified ReplicateStage to consume leader information from bank

* Restart RPC Services in Leader To Validator Transition

* Make VoteContract Context Free

* Remove voting from ClusterInfo and Tpu

* Remove dependency on ActiveValidators in LeaderScheduler

* Switch VoteContract to have two steps 1) Register 2) Vote. Change thin client to create + register a voting account on fullnode startup

* Remove check in leader_to_validator transition for unique references to bank, b/c jsonrpc service and rpcpubsub hold references through jsonhttpserver
2018-10-25 16:58:40 -07:00
carllin 0bd1412562
Switch leader scheduler to use PoH ticks instead of Entry height (#1519)
* Add PoH height to process_ledger()

* Moved broadcast_stage Leader Scheduling logic to use Poh height instead of entry_height

* Moved LeaderScheduler logic to PoH in ReplicateStage

* Fix Leader scheduling tests to use PoH instead of entry height

* Change is_leader detection in repair() to use PoH instead of entry height

* Add tests to LeaderScheduler for new functionality

* fix Entry::new and genesis block PoH counts

* Moved LeaderScheduler to PoH ticks

* Cleanup to resolve PR comments
2018-10-18 22:57:48 -07:00
Stephen Akridge 37a0b7b132 Initial validator code for rust side hooks for chacha cuda parallel encrypt 2018-10-18 13:50:19 -07:00
Stephen Akridge c30b605047 Actually submit the storage mining proof
Get an aidrop so replicator can submit mining transaction

Some other minor type cleanup.
2018-10-18 13:50:19 -07:00
Rob Walker d0f43e9934 consolidate tmp ledgers 2018-10-18 08:45:31 -06:00