Commit Graph

153 Commits

Author SHA1 Message Date
Greg Fitzgerald ec5a8141eb cargo fix --edition 2018-12-08 23:19:55 -07:00
Sathish 2de45a4da5
Update airdrop tokens to 3 for fullnode (#2051)
Filter out leader while computing the super majority stake
2018-12-08 16:54:42 -08:00
Greg Fitzgerald 0a83b17cdd
Upgrade to Rust 1.31.0 (#2052)
* Upgrade to Rust 1.31.0
* Upgrade nightly
* Fix all clippy warnings
* Revert relaxed version check and update
2018-12-07 20:01:28 -07:00
Greg Fitzgerald 97b1156a7a Rename Ncp to GossipService
And BroadcastStage to BroadcastService since it's not included in the
TPU pipeline.
2018-12-06 15:48:19 -07:00
Michael Vines c99f93e40a Remove signature.rs indirection 2018-12-03 13:50:00 -08:00
carllin 4ae58cc854
Change range of leader scheduler to match current broadcasts (#1920) 2018-12-03 00:10:43 -08:00
Sagar Dhawan 34c3a0cc1f
Add signature verification to gossip (#1937) 2018-12-01 12:00:30 -08:00
carllin 57a384d6a0
Rocks db window service (#1888)
* Add db_window module for windowing functions from RocksDb

* Replace window with db_window functions in window_service

* Fix tests

* Make note of change in db_window

* Create RocksDb ledger in bin/fullnode

* Make db_ledger functions generic

* Add db_ledger to bin/replicator
2018-11-24 19:32:33 -08:00
Rob Walker 6b910d1bd4 add tpu_forwarding, simplify ClusterInfo::new() from Result<Self> to Self 2018-11-19 20:45:49 -08:00
Rob Walker 3d113611cc
remove Result<> return from ClusterInfo::new() (#1869)
strip Result<> for ClusterInfo::new()
2018-11-19 11:25:14 -08:00
Michael Vines d96a6b42a5 Move drone into its own crate 2018-11-16 20:42:21 -08:00
Michael Vines 6ac5700f2e Move metrics into its own crate 2018-11-16 15:10:07 -08:00
Michael Vines a0dd8617be Remove airdrop from fullnode 2018-11-16 13:25:55 -08:00
anatoly yakovenko a41254e18c
Add scalable gossip library (#1546)
* Cluster Replicated Data Store

Separate the data storage and merge strategy from the network IO boundary.
Implement an eager push overlay for transporting recent messages.

Simulation shows fast convergence with 20k nodes.
2018-11-15 13:23:26 -08:00
Sagar Dhawan 729d28d910 Add poh verification before processing entries
- Replicate stage now verifies entries delivered
  by the window
- Minor refactor of entries_from_blobs
2018-11-13 14:17:00 -08:00
carllin 6335be803c
Broadcast last tick before leader rotation (#1766)
* Broadcast last tick before leader rotation to everybody on network

* Add test

* Refactor broadcast
2018-11-13 02:21:37 -08:00
Sagar Dhawan a77b1ff767
Revert "Migrate from ring to ed25519-dalek" (#1798)
* Revert "Migrate from ring to ed25519-dalek"

This reverts commit 7c610b216b.

* Fix test failures with revert
2018-11-12 22:34:43 -08:00
Greg Fitzgerald fea86b2955 No longer serialize as JSON-encoded pkcs8
That's supposed to be an ASCII format, but we're not making use
of it. We can switch back to that some day, but if we do, it shouldn't
be JSON-encoded.
2018-11-11 09:22:52 -07:00
Greg Fitzgerald 7c610b216b Migrate from ring to ed25519-dalek
Why?

* Pure Rust, no BoringSSL (or OpenSSL) dependency
* Those avx2 benchmarks
* ring includes far more than what we need
* ring author won't add release tags: https://github.com/briansmith/ring#versioning--stability
2018-11-11 09:22:52 -07:00
Rob Walker 6c10458b5b
leader slots in Blobs (#1732)
* add leader slot to Blobs
* remove get_X() methods in favor of X() methods for Blob
* add slot to get_scheduled_leader()
2018-11-07 13:18:14 -08:00
Greg Fitzgerald 8be7c13d2d
Stub out architecture book (#1674)
* Stub out architecture documentation

* Add book HTML generation and book tests to CI

* Add heading

* Better table of contents

* Reference existing documentation

Move ASCII art from code comments into rendered SVG

* Attempt to fix CI

* Add lamport docs

And truncate lines to 80 characters

* Fix links

And reference shorter, newer description of PoH.

* Replace ASCII art with SVG

* Streamline for Pillbox

* Update path before optional install

* Use $CARGO_HOME instead of $HOME

* Delete code

Attempt to describe all data structures without code.

* Boot RPU from docs, add JsonRpcService

Also, use Rust naming conventions in the block diagrams to
minimize the jump from docs to code.

* Latest code uses tick_height

* Rename bob/ folder to art/

A home for any ASCII art

* Import JSON RPC API

* More mdbook docs

* Add Ncp

* Cleanup links

* Move pipelining description into fullnode description

* Move high-level transaction docs into top-level doc

* Delete unused files
2018-11-06 18:00:58 -07:00
Tyera Eulberg d7ea66b6a1 RPC and Pubsub, bind to 0.0.0.0 2018-11-06 15:45:36 -07:00
Rob Walker 9b43b00d5c
remove tick_count, leader_scheduler, from broadcast code (#1725) 2018-11-06 13:17:41 -08:00
Tyera Eulberg f683817b48 Remove RPU; replace with RPC 2018-11-05 20:30:47 -07:00
Rob Walker 1fbf1d2cf2
Add checkpoint, rollback to to bank (#1662)
add linked-list capability to accounts

change accounts from a linked list to a VecDeque

add checkpoint and rollback for lastids

add subscriber notifications for rollbacks

checkpoint transaction count, too
2018-11-05 09:47:41 -08:00
carllin 298bd6479a
Add first leader to genesis (#1681)
* Add first leader to genesis entries, consume in genesis.sh

* Set bootstrap leader in the bank on startup, remove instantiation of bootstrap leader from bin/fullnode

* Remove need to initialize bootstrap leader in leader_scheduler, now can be read from genesis entries

* Add separate interface new_with_leader() in mint for creating genesis leader entries
2018-11-02 14:32:05 -07:00
Rob Walker 13bfdde228
remove ledger tail code, WINDOW_SIZE begone (#1617)
* remove WINDOW_SIZE, use window.window_size()
* move ledger tail, redundant with ledger-based repair
2018-10-30 10:05:18 -07:00
carllin f6c8e1a4bf
Vote contract (#1552)
* Add Vote Contract

* Move ownership of LeaderScheduler from Fullnode to the bank

* Modified ReplicateStage to consume leader information from bank

* Restart RPC Services in Leader To Validator Transition

* Make VoteContract Context Free

* Remove voting from ClusterInfo and Tpu

* Remove dependency on ActiveValidators in LeaderScheduler

* Switch VoteContract to have two steps 1) Register 2) Vote. Change thin client to create + register a voting account on fullnode startup

* Remove check in leader_to_validator transition for unique references to bank, b/c jsonrpc service and rpcpubsub hold references through jsonhttpserver
2018-10-25 16:58:40 -07:00
jackcmay 5ca52d785c
Preload BPF loader (#1573)
Preload BPF loader
2018-10-23 14:44:41 -07:00
Pankaj Garg 8d9912b4e2
Move ledger write to its own stage (#1577)
* Move ledger write to its own stage

- Also, rename write_stage to leader_vote_stage, as write functionality
  is moved to a different stage

* Address review comments

* Fix leader rotation test failure

* address review comments
2018-10-23 14:42:48 -07:00
carllin 0bd1412562
Switch leader scheduler to use PoH ticks instead of Entry height (#1519)
* Add PoH height to process_ledger()

* Moved broadcast_stage Leader Scheduling logic to use Poh height instead of entry_height

* Moved LeaderScheduler logic to PoH in ReplicateStage

* Fix Leader scheduling tests to use PoH instead of entry height

* Change is_leader detection in repair() to use PoH instead of entry height

* Add tests to LeaderScheduler for new functionality

* fix Entry::new and genesis block PoH counts

* Moved LeaderScheduler to PoH ticks

* Cleanup to resolve PR comments
2018-10-18 22:57:48 -07:00
Rob Walker d0f43e9934 consolidate tmp ledgers 2018-10-18 08:45:31 -06:00
Rob Walker 912eb5e8e9
remove bank.is_leader, dead code (#1516) 2018-10-16 15:26:44 -07:00
Tyera Eulberg 1b2e9122d5 Pubsub listen on random open port when rpc does (quiet some test errors) 2018-10-16 00:11:26 -06:00
Michael Vines 537436bd5e RPC PubSub now uses a well-known socket 2018-10-16 00:11:26 -06:00
Tyera Eulberg 1515bba9c6
Use cluster_info in rpc to get current leader addresses (#1480) 2018-10-12 14:25:56 -06:00
carllin 47f69f2d24
1) Switch broken tests to generate an empty tick in their ledgers to use as last_id, 2) Fix bug where PoH generator in BankingStage did not referenced the last tick instead of the last entry on startup, causing ledger verification to fail on the new tick added by the PoH generator (#1479) 2018-10-12 00:39:10 -07:00
Stephen Akridge 9dd4dc2088 Mark failing tests as ignore 2018-10-11 15:32:36 -07:00
carllin 9931ac9780
Leader scheduler plumbing (#1440)
* Added LeaderScheduler module and tests

* plumbing for LeaderScheduler in Fullnode + tests. Add vote processing for active set to ReplicateStage and WriteStage

* Add LeaderScheduler plumbing for Tvu, window, and tests

* Fix bank and switch tests to use new LeaderScheduler

* move leader rotation check from window service to replicate stage

* Add replicate_stage leader rotation exit test

* removed leader scheduler from the window service and associated modules/tests

* Corrected is_leader calculation in repair() function in window.rs

* Integrate LeaderScheduler with write_stage for leader to validator transitions

* Integrated LeaderScheduler with BroadcastStage

* Removed gossip leader rotation from crdt

* Add multi validator, leader test

* Comments and cleanup

* Remove unneeded checks from broadcast stage

* Fix case where a validator/leader need to immediately transition on startup after reading ledger and seeing they are not in the correct role

* Set new leader in validator -> validator transitions

* Clean up for PR comments, refactor LeaderScheduler from process_entry/process_ledger_tail

* Cleaned out LeaderScheduler options, implemented LeaderScheduler strategy that only picks the bootstrap leader to support existing tests, drone/airdrops

* Ignore test_full_leader_validator_network test due to bug where the next leader in line fails to get the last entry before rotation (b/c it hasn't started up yet). Added a test test_dropped_handoff_recovery go track this bug
2018-10-10 16:49:41 -07:00
Greg Fitzgerald 95701114e3 Crdt -> ClusterInfo 2018-10-09 03:49:39 -06:00
Carl 5491422b12 Fix validator_to_leader_transition test to not start up tpu after shutting down tvu, as the tpu now outputs ticks that will mess up the verification check 2018-10-03 21:58:29 -07:00
Pankaj Garg e10574c64d Remove recycler and it's usage
- The memory usage due to recycler was high, and incrementing with
  time.
2018-09-27 10:42:37 -06:00
jackcmay 9c47e022dc
break dependency of programs on solana core (#1371)
* break dependency of programs on Solana core
2018-09-27 07:49:26 -07:00
Greg Fitzgerald b7ae5b712a Move Pubkey into its own module 2018-09-26 20:40:40 -06:00
Anatoly Yakovenko 93c4f6c9b8 Synchronize PoH, bank last_id queue and ledger entry channel.
PoH, bank's last_id queue and the Entry channel need to have a synchronized order of ids.
2018-09-26 16:19:03 -07:00
carllin e7383a7e66
Validator to leader (#1303)
* Add check in window_service to exit in checks for leader rotation, and propagate that service exit up to fullnode

* Added logic to shutdown Tvu once ReplicateStage finishes

* Added test for successfully shutting down validator and starting up leader

* Add test for leader validator interaction

* fix streamer to check for exit signal before checking socket again to prevent busy leaders from never returning

* PR comments - Rewrite make_consecutive_blobs() function, revert genesis function change
2018-09-25 15:41:29 -07:00
Greg Fitzgerald 95677a81c5 Pacify clippy 2018-09-24 13:36:31 -06:00
Jack May a1f01fb8f8 revert is_some to not is_none, causes test failure 2018-09-23 17:09:18 -06:00
Jack May b2be0e2e5e fix clippy warning 2018-09-23 17:09:18 -06:00
Jack May 1a45587c08 fix clippy warnings 2018-09-23 17:09:18 -06:00