Commit Graph

163 Commits

Author SHA1 Message Date
Greg Fitzgerald 11f1c00ca7 Only send pubkey to ReplayStage 2019-01-30 21:59:05 -07:00
Rob Walker 1b50fbbc90
remove Result<> from Blob accessors, add parent (#2608)
* remove Result<> from Blob accessors, add parent
* update chacha's golden
* fixup benches
2019-01-30 20:18:28 -08:00
Tyera Eulberg 6da7a784f2
Stream entries (#2582)
* Add entry streaming option

* Fix tests

* Remove obsolete comment

* Move entry stream functionality to struct w/ trait in order to test without i/o
2019-01-29 00:21:27 -08:00
Stephen Akridge 53afa64634 Remove storage_state from the bank
Construct in TVU and pass to RPC and StorageStage instead.
2019-01-28 15:41:41 -08:00
Greg Fitzgerald 1bae87d4b3 Add unit-test-friendly VoteSignerProxy constructor 2019-01-26 14:56:49 -07:00
Sagar Dhawan d65e7b9fcc
Speedup rotation (#2468)
Speedup leader to validator transitions
2019-01-26 13:58:08 +05:30
Michael Vines 4bb6549895 Genesis block is now a json file 2019-01-25 09:05:15 -08:00
Tyera Eulberg b7be5b9a7a Add no-signer argument 2019-01-24 10:47:37 -08:00
Stephen Akridge abbb037888 Implement storage contract logic 2019-01-21 08:36:49 -08:00
Stephen Akridge 1fd7bd7ede Storage fixes
* replicators generate their sample values
* fixes to replicator block height logic
2019-01-18 13:05:35 -08:00
Pankaj Garg d7594b19fc
Implemented a trait for vote signer service (#2386)
* Implemented a trait for vote signer service

* removes need for RPC in unit tests for vote signing

* fix build errors

* address some review comments
2019-01-11 12:58:31 -08:00
Stephen Akridge 491bca5e4b Remove ledger.rs
Split into entry.rs for entry-constructing functions and EntrySlice
trait and db_ledger.rs for ledger helper test functions.
2019-01-09 15:15:47 -08:00
Greg Fitzgerald a82a5ae184 Delete unused code
The ignored test is still broken, but at least no longer creates a
window for no reason.

Also removed all remaining references to "ncp".
2019-01-08 14:09:50 -08:00
Greg Fitzgerald d2431128c7 Remove WriteStage from TPU/TVU diagrams
Fixes #2312
2019-01-08 08:42:06 -08:00
Pankaj Garg 91bd38504e
Use vote signer service in fullnode (#2009)
* Use vote signer service in fullnode

* Use native types for signature and pubkey, and address other review comments

* Start local vote signer if a remote service address is not provided

* Rebased to master

* Fixes after rebase
2019-01-05 12:57:52 -08:00
Michael Vines 1f6346d880 De-dup ledgers - db_ledger is now the only ledger written to disk 2019-01-04 16:37:00 -08:00
Sagar Dhawan 0bea870b22
Dynamic N layer 'avalanche' broadcast and retransmit (#2058)
* Dynamic N layer avalanche broadcast and retransmit
2019-01-02 14:16:15 +05:30
Michael Vines 58f2598d5d Revert "Validators make a transaction to advertise their storage last_id"
This reverts commit a1759aed19.
2018-12-23 14:02:09 -08:00
Stephen Akridge a1759aed19 Validators make a transaction to advertise their storage last_id
* Also implement more storage contract logic
* Add transactions for proof validation,
* Move storage state members into system storage account userdata
2018-12-21 15:45:30 -08:00
Michael Vines 37d7ad819b Purge DB::destroy() usage 2018-12-20 10:38:03 -08:00
Rob Walker a65022aed7
DbLedger doesn't need to be mut, doesn't need an RwLock (#2215)
* DbLedger doesn't need to be mut, doesn't need an RwLock

* fix erasure cases
2018-12-18 15:18:57 -08:00
Michael Vines 6ac466c0a4 Move src/logger.rs into logger/ crate to unify logging across the workspace 2018-12-14 13:10:43 -08:00
Stephen Akridge 7cdbbfa88e Storage stage updates
* Remove logging init from storage program: saw a crash in a test
  indicating the logger being init'ed twice.
* Add entry_height mining proof to indicate which segment the result is
  for
* Add an interface to get storage miner pubkeys for a given entry_height
* Add an interface to get the current storage mining entry_height
* Set the tvu socket to 0.0.0.0:0 in replicator to stop getting entries
  after the desired ledger segment is downloaded.
* Use signature of PoH height to determine which block to download for
  replicator.
2018-12-13 11:30:12 -08:00
Carl 245362db96 Make a dummy version of serving repairs from db_ledger 2018-12-12 19:12:51 -08:00
Mark b1b190b80d Fix too many args in Tvu::new (#2114)
* Reduce args in Tvu::new under to 8

Now pass in sockets through a the crate::tvu::Sockets struct

Move ClusterInfo.keypair to pub(crate) in order to remove redundant
signing keypair parameter

* remove commented code
2018-12-12 18:57:48 -08:00
Rob Walker aeee25e703
add tick_height to Entry to be able to repair by period, chain forks of Entries, etc. (#2096) 2018-12-10 20:03:04 -08:00
Greg Fitzgerald 5e703dc70a Free up the term 'replicate' for exclusive use in replicator
Also, align Sockets field names with ContactInfo.
2018-12-10 15:26:43 -07:00
Greg Fitzgerald a8d6c75a24 cargo +nightly fix --features=bpf_c,cuda,erasure,chacha --edition-idioms 2018-12-08 23:19:55 -07:00
Greg Fitzgerald ec5a8141eb cargo fix --edition 2018-12-08 23:19:55 -07:00
Greg Fitzgerald 0a83b17cdd
Upgrade to Rust 1.31.0 (#2052)
* Upgrade to Rust 1.31.0
* Upgrade nightly
* Fix all clippy warnings
* Revert relaxed version check and update
2018-12-07 20:01:28 -07:00
Stephen Akridge 3441d3399b Replicator rework
* Move more of the replicator logic into the replicator class
* Add support for the RPC interface to query the storage last_id value
  that the replicator would sign and use to pick a block.
* Fix replicator connecting to gossip and change test to exercise that
  scenario.
2018-12-07 15:20:36 -08:00
Greg Fitzgerald 97b1156a7a Rename Ncp to GossipService
And BroadcastStage to BroadcastService since it's not included in the
TPU pipeline.
2018-12-06 15:48:19 -07:00
Michael Vines 42689d4842 cargo fmt 2018-12-05 10:49:06 -08:00
Michael Vines 214ed3667c Move system_transaction out of src/ 2018-12-05 10:49:06 -08:00
Michael Vines c99f93e40a Remove signature.rs indirection 2018-12-03 13:50:00 -08:00
Michael Vines 0878bd53d9 Delete stub src/transaction.rs 2018-11-29 23:07:57 -08:00
carllin 57a384d6a0
Rocks db window service (#1888)
* Add db_window module for windowing functions from RocksDb

* Replace window with db_window functions in window_service

* Fix tests

* Make note of change in db_window

* Create RocksDb ledger in bin/fullnode

* Make db_ledger functions generic

* Add db_ledger to bin/replicator
2018-11-24 19:32:33 -08:00
Rob Walker 3d113611cc
remove Result<> return from ClusterInfo::new() (#1869)
strip Result<> for ClusterInfo::new()
2018-11-19 11:25:14 -08:00
Michael Vines d96a6b42a5 Move drone into its own crate 2018-11-16 20:42:21 -08:00
anatoly yakovenko a41254e18c
Add scalable gossip library (#1546)
* Cluster Replicated Data Store

Separate the data storage and merge strategy from the network IO boundary.
Implement an eager push overlay for transporting recent messages.

Simulation shows fast convergence with 20k nodes.
2018-11-15 13:23:26 -08:00
Jack May 4e553ea095 test_replicate fails locally, ignore 2018-11-13 17:13:25 -08:00
Sagar Dhawan 729d28d910 Add poh verification before processing entries
- Replicate stage now verifies entries delivered
  by the window
- Minor refactor of entries_from_blobs
2018-11-13 14:17:00 -08:00
Greg Fitzgerald 2a6046de8e
Cleanup TVU code to look like its block diagram (#1737)
* Reorg TVU code to look like TVU diagram

And move channel creation into LedgerWriteStage so that it can
be used in the same was as all the other stages.

* Delete commented out code
2018-11-07 19:25:36 -07:00
Rob Walker 6c10458b5b
leader slots in Blobs (#1732)
* add leader slot to Blobs
* remove get_X() methods in favor of X() methods for Blob
* add slot to get_scheduled_leader()
2018-11-07 13:18:14 -08:00
Greg Fitzgerald 8be7c13d2d
Stub out architecture book (#1674)
* Stub out architecture documentation

* Add book HTML generation and book tests to CI

* Add heading

* Better table of contents

* Reference existing documentation

Move ASCII art from code comments into rendered SVG

* Attempt to fix CI

* Add lamport docs

And truncate lines to 80 characters

* Fix links

And reference shorter, newer description of PoH.

* Replace ASCII art with SVG

* Streamline for Pillbox

* Update path before optional install

* Use $CARGO_HOME instead of $HOME

* Delete code

Attempt to describe all data structures without code.

* Boot RPU from docs, add JsonRpcService

Also, use Rust naming conventions in the block diagrams to
minimize the jump from docs to code.

* Latest code uses tick_height

* Rename bob/ folder to art/

A home for any ASCII art

* Import JSON RPC API

* More mdbook docs

* Add Ncp

* Cleanup links

* Move pipelining description into fullnode description

* Move high-level transaction docs into top-level doc

* Delete unused files
2018-11-06 18:00:58 -07:00
Tyera Eulberg f683817b48 Remove RPU; replace with RPC 2018-11-05 20:30:47 -07:00
Rob Walker 1fbf1d2cf2
Add checkpoint, rollback to to bank (#1662)
add linked-list capability to accounts

change accounts from a linked list to a VecDeque

add checkpoint and rollback for lastids

add subscriber notifications for rollbacks

checkpoint transaction count, too
2018-11-05 09:47:41 -08:00
Stephen Akridge cf00354f42 Add storage stage which does storage mining verification for validators 2018-10-31 13:44:21 -07:00
carllin f6c8e1a4bf
Vote contract (#1552)
* Add Vote Contract

* Move ownership of LeaderScheduler from Fullnode to the bank

* Modified ReplicateStage to consume leader information from bank

* Restart RPC Services in Leader To Validator Transition

* Make VoteContract Context Free

* Remove voting from ClusterInfo and Tpu

* Remove dependency on ActiveValidators in LeaderScheduler

* Switch VoteContract to have two steps 1) Register 2) Vote. Change thin client to create + register a voting account on fullnode startup

* Remove check in leader_to_validator transition for unique references to bank, b/c jsonrpc service and rpcpubsub hold references through jsonhttpserver
2018-10-25 16:58:40 -07:00
Pankaj Garg 8d9912b4e2
Move ledger write to its own stage (#1577)
* Move ledger write to its own stage

- Also, rename write_stage to leader_vote_stage, as write functionality
  is moved to a different stage

* Address review comments

* Fix leader rotation test failure

* address review comments
2018-10-23 14:42:48 -07:00