Commit Graph

205 Commits

Author SHA1 Message Date
Tyera Eulberg 9ca2f5b3f7 Move all handling except network/gossip from /bin to wallet module 2018-09-14 08:21:33 -06:00
Pankaj Garg 90df6237c6
Implements recvmmsg() for UDP packets (#1161)
* Implemented recvmmsg() for UDP packets

- This change implements binding between libc API for recvmmsg()
- The function can receive multiple packets using one system call

Fixes #1141

* Added unit tests for recvmmsg()

* Added recv_mmsg() wrapper for non Linux OS

* Address review comments for recvmmsg()

* Remove unnecessary imports

* Moved target specific dependencies to the function
2018-09-13 14:41:28 -07:00
Greg Fitzgerald 7f669094de Split window into two modules 2018-09-10 11:40:26 -06:00
Rob Walker 97c3125a78
improve localnet-sanity's robustness (#1160)
* fix poll_gossip_for_leader() loop to actually wait
         for 30 seconds
    * reduce reuseaddr use to only when necessary,
         try to avoid already bound sockets
    * move nat.rs to netutil.rs
    * add gossip tracing to thin_client and bench-tps
2018-09-09 04:50:43 +09:00
Pankaj Garg 05460eec0d
Open multiple sockets for transaction UDP port (#1128)
* Reuse UDP port and open multiple sockets for transaction address

* Fixed failing crdt tests

* Add tests for reusing UDP ports

* Address review comments

* Updated bench-streamer to use multiple receive sockets

* Fix minimum number of recv sockets for bench-streamer

* Address review comments

Fixes #1132

* Moved bind_to function to nat.rs
2018-09-06 14:13:40 -07:00
Greg Fitzgerald 3ab12076e8 Convert voting functions to methods
More idiomatic Rust.
2018-09-04 05:53:58 -10:00
Greg Fitzgerald fda4523cbf Fix broken doc 2018-08-23 13:42:54 -07:00
Greg Fitzgerald 2d35345c50 Boot unused creates 2018-08-20 16:48:37 -06:00
Tyera Eulberg d3c09b4e96 Update jsonrpc dependency syntax 2018-08-18 12:29:49 -06:00
Tyera Eulberg fc1dbddd93 Implement json-rpc functionality 2018-08-18 12:29:49 -06:00
Greg Fitzgerald 0e66606c7f Rename broadcaster to broadcast_stage
And move retransmitter code into retransmit_stage.

TODO: Add a BroadcastStage service
2018-08-13 20:17:16 -06:00
Greg Fitzgerald 2727067b94 Move winow into its own module 2018-08-13 20:17:16 -06:00
Greg Fitzgerald 6a8a494f5d Rename WindowStage to RetransmitStage
The window is used for both broadcasting from leader to validator
and retransmitting between validators.
2018-08-13 20:17:16 -06:00
Greg Fitzgerald a09d2e252a Move window dependencies out of streamer
No tests!?
2018-08-13 20:17:16 -06:00
Stephen Akridge a206f2570d Add hostname to metrics on panic 2018-08-10 17:08:40 -07:00
Tyera Eulberg a86618faf3 Add PublicKey wrapper
Add custom formatting for PublicKey display and debug
2018-08-01 16:15:14 -07:00
sakridge 2ea6f86199 Submit leader's vote after observing 2/3 validator votes (#780)
* fixup!

* fixups!

* send the vote and count it

* actually vote

* test

* Spelling fixes

* Process the voting transaction in the leader's bank

* Send tokens to the leader

* Give leader tokens in more cases

* Test for write_stage::leader_vote

* Request airdrop inside fullnode and not the script

* Change readme to indicate that drone should be up before leader

And start drone before leader in snap scripts

* Rename _kp => _keypair for keypairs and other review fixups

* Remove empty else
* tweak test_leader_vote numbers to be closer to testing 2/3 boundary
* combine creating blob and transaction for leader/validator
2018-07-31 22:07:53 -07:00
Michael Vines 7672506b45 Validators now vote once a second regardless 2018-07-26 17:07:42 -07:00
Anatoly Yakovenko be2bf69c93 initial vote stage
wip

voting

wip

move voting into the replicate stage

update

fixup!

fixup!

fixup!

fixup!

fixup!

fixup!

fixup!

fixup!

fixup!

fixup!

update

fixup!

fixup!

fixup!

tpu processing votes in entries before write stage

fixup!

fixup!

txs

make sure validators have an account

fixup!

fixup!

fixup!

exit fullnode correctly

exit on exit not err

try 50

add delay for voting

300

300

startup logs

par start

100

no rayon

retry longer

log leader drop

fix distance

50 nodes

100

handle deserialize error

update

fix broadcast

new table every time

tweaks

table

update

try shuffle table

skip kill

skip add

purge test

fixed tests

rebase 2

fixed tests

fixed rebase

cleanup

ok for blobs to be longer then window

fix init window

60 nodes
2018-07-10 13:32:31 -06:00
Michael Vines 22c0e3cd54 Metrics v0.1 2018-07-07 19:40:09 -07:00
Greg Fitzgerald 77bf17064a Add Service trait
Added a consistent interface to all the microservices.
2018-07-04 16:40:34 -06:00
Anatoly Yakovenko b8fe5ae076 rename server to fullnode 2018-07-02 16:34:49 -06:00
Anatoly Yakovenko fa247196c0 fullnode lib 2018-07-02 16:34:49 -06:00
Michael Vines 0b56d603c2 Client NAT traversal 0.1
UPnP is now used to request a port on the NAT be forwarded to the local machine.
This obviously only works for NATs that support UPnP, and thus is not a panacea
for all NAT-related connectivity issues.

Notable hacks in this patch include a transmit/receive UDP socket pair to work
around current protocol limitations whereby the full node assumes its peer can
receive on the same UDP port it transmitted from.
2018-06-29 17:36:26 -07:00
Greg Fitzgerald c4b62e19f2 Do Proof of History verification before appending entries to the bank
Note: replicate_stage is still using `process_entries()` because
changing it to `process_blocks()` causes the `test_replicate` test to
fail.
2018-06-29 15:35:39 -06:00
Greg Fitzgerald 05edfad13a Fix compiler warnings 2018-06-26 15:03:15 -07:00
OEM Configuration (temporary user) da3bb6fb93 ran linter 2018-06-26 12:31:04 -06:00
OEM Configuration (temporary user) 08bcb62016 added remote table to update respones 2018-06-26 12:31:04 -06:00
Greg Fitzgerald c7ce454188 Use pnet_datalink instead of all of pnet
pnet_transport takes a long time to build. It's been especially
painful from within a docker container for reasons I don't care
to understand. pnet_datalink is the only part of pnet we're using
so booting the rest.
2018-06-24 10:39:59 -06:00
Tyera Eulberg e57dae0f31 Update config and dependencies for solana-drone 2018-06-22 14:52:36 -07:00
anatoly yakovenko c24b0a1a3f
TVU rework (#352)
Refactored TVU, into stages
* blob fetch stage for blobs
* window stage for maintaining the blob window
* pulled out NCP out of the TVU so they can be separate units
TVU is now just the fetch -> window -> request and bank processing
2018-06-13 21:52:23 -07:00
Greg Fitzgerald 7aa05618a3 data_replicator -> ncp
Fixes #327
2018-06-07 17:11:17 -06:00
Anatoly Yakovenko c2a9395a4b perf counters 2018-06-07 14:59:21 -07:00
Greg Fitzgerald 8bd10e7c4c Cleanup top-level lib doc 2018-06-07 15:24:44 -06:00
Greg Fitzgerald ee44e51b30 Better docs for the bank 2018-06-07 09:25:36 -06:00
Greg Fitzgerald 359a8397c0 Make bootstrapping functions accessible to other binaries 2018-05-31 16:15:02 -06:00
Greg Fitzgerald c9fd5d74b5 Boot futures 0.1
We added them thinking it'd be a good stepping stone towards an
asynchronous thin client, but it's used inconsistently and where
it used, the function is still synchronous, which is just confusing.
2018-05-31 14:13:09 -06:00
Anatoly Yakovenko cef1c208a5 Crdt pipeline, coalesce window repair requests in the listener by examining all of them at once, and ublock those threads from doing io. 2018-05-30 14:04:48 -06:00
Greg Fitzgerald ac0791826a plan.rs -> payment_plan.rs 2018-05-29 18:09:03 -06:00
Greg Fitzgerald f82cbf3a27 Move Budget EDSL into its own module 2018-05-29 18:09:03 -06:00
Greg Fitzgerald f2ccc133a2 Finally made fetch happen 2018-05-29 14:04:27 -06:00
Greg Fitzgerald 46c19a5783 Rename sigverify modules 2018-05-25 17:31:07 -06:00
Greg Fitzgerald 4cdf873f98 Delete event.rs 2018-05-25 16:47:21 -06:00
Anatoly Yakovenko 021953d59a cleanup 2018-05-22 15:30:46 -07:00
Anatoly Yakovenko bbe89df2ff fmt 2018-05-22 15:18:07 -07:00
Anatoly Yakovenko a638ec5911 builds 2018-05-22 15:17:59 -07:00
Greg Fitzgerald cfe8b3fc55 Wrap the RPU with new object Server 2018-05-15 11:00:01 -06:00
Greg Fitzgerald ef6bd7e3b8 Add TPU 2018-05-14 17:36:19 -06:00
Greg Fitzgerald 374bff6550 Extract event processing from request_stage 2018-05-14 17:31:27 -06:00
Greg Fitzgerald 81706f2d75 Move write_service and drain_service into new write_stage module 2018-05-14 16:31:31 -06:00
Greg Fitzgerald d2dd005a59 accountant -> bank 2018-05-14 15:33:11 -06:00
Greg Fitzgerald 6e8f99d9b2 Purge EventProcessor 2018-05-14 14:45:29 -06:00
Greg Fitzgerald 17cc9ab07f Rename Historian to RecordStage
Historian was a legacy name. The new name reflects the new pipelined
architecture.
2018-05-14 14:19:19 -06:00
Greg Fitzgerald aec05ef602 Move RequestProcessor into its own module 2018-05-12 17:50:55 -06:00
Greg Fitzgerald e5d46d998b Move thin client messages into their own module 2018-05-12 17:41:27 -06:00
Greg Fitzgerald a3d2831f8c Free up the name 'accounting_stage' 2018-05-12 14:05:57 -06:00
Greg Fitzgerald 421d9aa501 Free up the name 'tpu' 2018-05-12 10:53:25 -06:00
Greg Fitzgerald 898f4971a2 Free up name 'thin_client_service' 2018-05-12 10:50:22 -06:00
Greg Fitzgerald 7ab3331f01 Move validation processor to its own module 2018-05-12 00:31:32 -06:00
Greg Fitzgerald ca80bc33c6 Move the writer stage's utilities to its own module 2018-05-11 22:36:16 -06:00
Greg Fitzgerald 19607886f7 Move sig verification stage into its own module 2018-05-11 21:51:37 -06:00
Greg Fitzgerald 43cd631579 Add thin_client_service 2018-05-09 14:56:34 -06:00
Greg Fitzgerald 876c77d0bc Extract accounting stage code from tpu 2018-05-09 09:22:46 -06:00
Greg Fitzgerald 1dca17fdb4 cargo +nightly fmt 2018-05-08 18:59:01 -06:00
Greg Fitzgerald 785e971698 AccountantSkel -> Tpu
The terms Stub and Skel come from OMG IDL and only made sense while
the Stub was acting as an RPC client for the the Accountant object.
Nowadays, the Stub interface looks nothing like the Accountant and
meanwhile we've recognized the multithreaded implementation is more
reminiscent of a pipelined CPU. Thus, we finally bite the bullet and
rename our modules.

AccountantSkel -> Tpu
AccountantStub -> ThinClient

Up next will be moving much of the TPU code into separate modules,
each representing a stage of the pipeline. The interface of each
will follow the precedent set by the Historian object.
2018-05-08 17:40:02 -06:00
Stephen Akridge bd0671e123 Rework sig processing threads and add perf for process/verify 2018-05-08 11:49:29 -07:00
Anatoly Yakovenko c2e2960bf7 Add broadcast impl 2018-05-03 10:34:01 -07:00
Robert Kelly 34d1805b54 Add FutureResult to return a Future that immediately resolves 2018-04-25 19:23:24 -07:00
Anatoly Yakovenko 3d7969d8a2 initial crdt implementation 2018-04-23 23:06:28 -07:00
rleungx 3da1fa4d88 improve the error messages 2018-04-21 21:52:55 +08:00
Stephen Akridge dea5ab2f79 Add erasure rust logic under feature flag 2018-04-18 19:42:09 -07:00
Anatoly Yakovenko 4944c965e4 update
heap

update

update

wip

use a vec and sort

builds

update

tests

update

fmt

update

progress

fmt

passes needs retransmit test

tests

cleanup

update

update

update

update

fmt
2018-04-16 20:33:09 -07:00
Greg Fitzgerald 5fc81dd6c8 Fix the nightly build
Nightly uses a different (but backward compatible) version of rustfmt.
2018-04-05 22:39:29 -06:00
Stephen Akridge f4466c8c0a Change for cuda verify integration 2018-04-05 20:00:44 -07:00
Anatoly Yakovenko 5ac7df17f9 Implement window service
Batch out of order blobs until we have a contigious window.
2018-04-03 13:53:19 -07:00
Greg Fitzgerald ddb21d151d Nightly rustfmt
Format code with the nightly version of rustfmt, which sorts imports.
2018-03-26 22:03:28 -06:00
Greg Fitzgerald 9238ee9572 No longer rename log crate 2018-03-19 10:18:51 -06:00
Greg Fitzgerald 64af37e0cd logger -> recorder
Free up namespace for a traditional runtime logger.
2018-03-19 10:16:21 -06:00
Greg Fitzgerald 9f9b79f30b log -> ledger
Free up namespace for traditional runtime logs.
2018-03-19 10:09:19 -06:00
Greg Fitzgerald e7da083c31 Move spending plans to their own crate 2018-03-17 19:56:15 -06:00
Anatoly Yakovenko 0135971769 Fast UdpSocket reader
* message needs to fit into 256 bytes
* allocator to keep track of blocks of messages
* udp socket receiver server that fills up the block as fast as possible
* udp socket sender server that sends out the block as fast as possible
2018-03-10 21:09:23 -06:00
Greg Fitzgerald 2379792e0a Add DateTime and Cancel conditions
Fixes #32, #33
2018-03-08 08:17:08 -07:00
Greg Fitzgerald 5dd567deef Rename Genesis to Mint
Genesis is a story of creation. We should only use that term to
for the event log that bootstraps the system.
2018-03-07 17:08:15 -07:00
Greg Fitzgerald 1436bb1ff2 Move entry into its own module
Hmm, Logger doesn't depend on log.
2018-03-06 17:40:01 -07:00
Greg Fitzgerald 5a44c36b1f Move hash into its own module 2018-03-06 17:40:01 -07:00
Greg Fitzgerald 624c151ca2 Add signature module
Because things other than transactions can be signed.
2018-03-06 12:48:28 -07:00
Greg Fitzgerald 4fcd9e3bd6 Give Transaction its own module 2018-03-06 12:18:17 -07:00
Greg Fitzgerald a4336a39d6 Initialize the testnode from a log
$ cargo run --bin silk-genesis-file-demo > demo-genesis.json
$ cat demo-genesis.json | cargo run --bin silk-genesis-block > demo-genesis.log
$ cat demo-genesis.log | cargo run --bin silk-testnode
2018-03-05 15:34:44 -07:00
Greg Fitzgerald 11a0f96f5e Add command-line tool for generating a genesis block 2018-03-03 17:35:05 -07:00
Greg Fitzgerald bebba7dc1f Give logger its own crate 2018-03-03 14:24:32 -07:00
Greg Fitzgerald 1b77e8a69a Move Event into its own crate
The log crate was starting to be the catch-all for all things
related to entries, events, signatures, and hashes. This split
shows us that:

* Event depends only on signatures, not on hashes [directly]
* All event testing was done via log testing (shame on me)
* Accounting depends only on events
2018-03-02 08:43:57 -07:00
Greg Fitzgerald 8299bae2d4 Add accountant stub 2018-02-28 16:01:12 -07:00
Greg Fitzgerald 604ccf7552 Add network interface for accountant 2018-02-28 14:00:04 -07:00
Greg Fitzgerald 5ca0ccbcd2 Add accountant 2018-02-27 10:54:06 -07:00
Greg Fitzgerald 0443b39264 Allow event log to hold events of any serializable (hashable) type 2018-02-26 16:42:31 -07:00
Greg Fitzgerald 08e501e57b Extend the event log with a Claim event to claim possession
Unlike a Discovery event, a Claim event associates a public key
with a hash. It's intended to to be used to claim ownership of
some hashable data. For example, a graphic designer could claim
copyright by hashing some image they created, signing it with
their private key, and publishing the hash-signature pair via
the historian. If someone else tries to claim it as their own,
the designer can point to the historian's log as cryptographically
secure evidence that the designer's copy existed before anyone
else's.

Note there's nothing here that verifies the first claim is the actual
content owner, only that the first claim almost certainly happened
before a second.
2018-02-24 10:09:49 -07:00
Greg Fitzgerald fa4e232d73 Add serialization/deseriation support to event log
See bincode and serde_json for usage:
https://github.com/TyOverby/bincode

Fixes #1
2018-02-20 16:26:13 -07:00
Greg Fitzgerald d97112d7f0 Explain proof-of-history in the readme
Also:
* Hash userdata so that verification works as the readme describes.
* Drop itertools package. Found a way to use std::iter instead.

Fixes #8
2018-02-20 14:04:49 -07:00
Greg Fitzgerald 0297edaf1f Use sha256 hashes instead of Rust's builtin hasher.
Causes a 20x performance degradation. Enabling asm did not
speed things up.
2018-02-19 16:23:53 -07:00
Greg Fitzgerald fd209ef1a9 Rename event crate to log 2018-02-18 09:59:33 -07:00
Greg Fitzgerald 831e2cbdc9 Add historian
A microservice that continuously generates hashes, only stopping to
tag messages with the latest hash.

Fixes #8
2018-02-17 20:58:23 -07:00
Greg Fitzgerald a74540470a Rename Tick to Event
* Define a tick to be an event with no user data.
* Use the term "event log" for now.
** Reserve the word "entry" for hash entries, and "item" for array items.
** Reserve the word "blockchain" for when the event is a block of something.
** Reserve the word "ledger" for when the event is of a particular type,
   such as transactions.
2018-02-15 10:13:59 -07:00
Greg Fitzgerald 31d79727b6 Parallelize verification 2018-02-14 15:23:13 -07:00
Greg Fitzgerald d23fda1936 First go at Proof of History 2018-02-14 11:00:01 -07:00
Greg Fitzgerald bad95b7712 cargo init 2018-02-14 07:23:59 -07:00