Commit Graph

2361 Commits

Author SHA1 Message Date
Pankaj Garg 637f58364a remove io from the tests 2019-01-28 13:52:13 -08:00
Pankaj Garg 1bd04b26e5 Remove ignore flag from rpc_pubsub tests 2019-01-28 13:52:13 -08:00
Michael Vines 29ef9370a6 Remove LeaderSchedulerConfig options 2019-01-28 13:51:01 -08:00
Stephen Akridge 2262f279d5 Reduce boilerplate code with helper function to create
fullnode/bank/genesis
2019-01-28 13:48:58 -08:00
Michael Vines e4f477cf90 Retype num_ticks as u64 to reduce casting 2019-01-28 11:24:50 -08:00
Greg Fitzgerald 33f921235d Improve message-signing ergonomics 2019-01-26 14:57:22 -07:00
Greg Fitzgerald 1bae87d4b3 Add unit-test-friendly VoteSignerProxy constructor 2019-01-26 14:56:49 -07:00
Greg Fitzgerald 1e43fb587e Rename the module that now contains only GenKeys 2019-01-26 06:57:24 -08:00
Sagar Dhawan d65e7b9fcc
Speedup rotation (#2468)
Speedup leader to validator transitions
2019-01-26 13:58:08 +05:30
Michael Vines 4bb6549895 Genesis block is now a json file 2019-01-25 09:05:15 -08:00
Rob Walker e9e01557b7 fix leaked threads from unclosed fullnode 2019-01-25 03:02:49 -08:00
Sathish e0f046b7a5
Optimize Transaction/Instruction serialization with custom routine (#2515)
* Optimize transaction serialization with custom routine to reduce the serialized size.

* Update serialized_size to accept self as parameter

* Optimize serialize / deserialize operations
2019-01-24 21:14:15 -08:00
Pankaj Garg 81c82b5af9
Add test for ignore ProgramErrors in process_entries (#2544) 2019-01-24 13:37:12 -08:00
Pankaj Garg a9b083e585
Set fetch stage socket non blocking to false while during recv (#2542)
* Set fetch stage socket non blocking to false while during recv

* remove ProgramError changes from this PR
2019-01-24 12:46:40 -08:00
Tyera Eulberg b7be5b9a7a Add no-signer argument 2019-01-24 10:47:37 -08:00
Greg Fitzgerald aba9df8457 Remove get_stake placeholder 2019-01-23 17:03:20 -08:00
Michael Vines 27c10d4468
cargo fmt 2019-01-22 21:56:04 -08:00
Stephen Akridge a17467aefd Lower level of message from storage_stage 2019-01-22 21:23:10 -08:00
Rob Walker 965dbbe835
stop enumeration if next entry is disjoint, band-aid (#2518)
* stop enumeration if next entry is disjoint, band-aid, fies #2426
* clippy
2019-01-22 15:50:36 -08:00
Michael Vines 122a5b2f69
dedup the 2019-01-22 09:47:43 -08:00
Rob Walker c84b796e17
remove dead code (#2512) 2019-01-21 16:24:11 -08:00
Pankaj Garg 7204bb40bf
Don't fail process_entries with ProgramErrors (#2509) 2019-01-21 15:26:06 -08:00
Mark f37eb533f1
Replicator timeout (#2480)
* Add timeout to Replicator::new; used when polling for leader

* Add timeout functionality to replicator ledger download

Shares the same timeout as polling for leader

Defaults to 30 seconds

* Add docs for Replicator::new
2019-01-21 15:37:41 -06:00
Michael Vines 6e8b69fc88 Cleanup leader_addr, it's really entrypoint_addr 2019-01-21 13:06:30 -08:00
Tyera Eulberg 35764225ed Remove socket from rpc test and move integration test 2019-01-21 12:29:04 -07:00
Rob Walker e6030d66eb
split load+execute from commit in bank, insert record between them in TPU code (#2487)
* split load+execute from commit in bank, insert record between them in TPU code
* clippy
* remove clear_signatures() race with commit_transactions()
* add #[test] back
2019-01-21 10:17:04 -08:00
Pankaj Garg 6611188edf
Move subscriptions to rpc_pubsub (#2490)
* Move subscriptions to rpc_pubsub

- this helps avoid recreating pubsub_service on node's role change

* fixed tests and addressed review comments

* fix clippy errors

* address review comments
2019-01-21 09:59:09 -08:00
Stephen Akridge abbb037888 Implement storage contract logic 2019-01-21 08:36:49 -08:00
Michael Vines 132d59ca6a new_bank_from_db_ledger need not be public 2019-01-21 08:11:13 -08:00
Michael Vines 5b73a8eceb Rework fullnode boot sequence 2019-01-19 21:35:46 -08:00
Michael Vines 2073188345 Fullnode no longer fails to process a ledger with ProgramErrors in it 2019-01-18 21:06:50 -08:00
Stephen Akridge 2f1107ff4f Add randomness to broadcast 2019-01-18 13:05:35 -08:00
Stephen Akridge 1fd7bd7ede Storage fixes
* replicators generate their sample values
* fixes to replicator block height logic
2019-01-18 13:05:35 -08:00
Michael Vines c0c38463c7 Remove hard coded ports 2019-01-17 23:34:21 -08:00
Michael Vines c1e142d1dc Revert "test_rpc_new fails locally, ignore for now"
This reverts commit 0c46f15f94.
2019-01-17 23:34:21 -08:00
Rob Walker 1f87d9ba4a
add bloom benchmarking, perf improvement from Fnv ~= 8X (#2477)
* add bloom benchmarking, perf improvement from Fnv ~= 8X
* have a look at bits.set()
* ignore new benches to pacify CI (solana_upload_perf?)
2019-01-17 18:22:21 -08:00
Michael Vines e74574706e Record Transactions with program errors in the ledger, they paid the fee 2019-01-17 13:55:56 -08:00
Stephen Akridge b296a9a0c7 Rename slice to segment to match book terminology 2019-01-17 10:19:45 -08:00
Michael Vines a84b6bc7e4 Overhaul wallet rpc/drone command-line arguments 2019-01-17 08:36:05 -08:00
Stephen Akridge 8e67a18551 Add network timeouts and fix tests
test_rpc_send_tx could fail if it didn't succeed on the first try
* add some retries to consistently pass
2019-01-16 15:59:40 -08:00
Michael Vines 526344c9ac Log signature status uniformly 2019-01-16 12:17:46 -08:00
Michael Vines f8bd19f5db Log the time it look to process the ledger for easier log inspection 2019-01-16 10:45:47 -08:00
Pankaj Garg 03d4d1cb36
Store and resend votes if leader's TPU port is unknown (#2438)
* Store and resend votes if leader's TPU port is unknown

* fix build errors

* fix failing tests
2019-01-16 06:14:55 -08:00
Michael Vines 4369c1a113 RPC port is no longer reset on leader-to-validator transition 2019-01-15 16:06:56 -08:00
Michael Vines b1e57e2a30 Retry rpc requests on connection failures
Applied a blanket default retry count of 5, which seems like enough but
not excessive retries.
2019-01-15 15:30:10 -08:00
Rob Walker 45c247fa5b
bloom for forking (#2431)
* bloom for forking
* clippy fixes
* remove bloom_hash_index
2019-01-15 13:56:54 -08:00
Pankaj Garg fa4608a95d
Change leader rotation time to a multiple of ticks per block (#2414)
* Change leader rotation time to a multiple of ticks per block

* fix component dependencies

* review comments
2019-01-15 12:07:58 -08:00
Rob Walker 022a97da99
revert revert of kill window (#2427)
* remove window code from most places
* window used only for testing
* remove unecessary clippy directives
2019-01-15 10:51:53 -08:00
Michael Vines a33c76a456 Remove JsonRpcRequestProcessor dependency 2019-01-14 17:39:31 -08:00
Michael Vines 11b1bd278a Remove unused exit field 2019-01-14 17:39:12 -08:00
Michael Vines e3a96ed3fc Minor new cleanup 2019-01-14 16:04:29 -08:00
Rob Walker 447243f994
Revert "remove window code from most places" (#2417)
* Revert "Fix link to book in Local Testnet section (#2416)"

This reverts commit 710c0c9980.

* Revert "Add current leader information to dashboard (#2413)"

This reverts commit f0300c1711.

* Revert "remove window code from most places (#2389)"

This reverts commit e3c0bd5a3f.
2019-01-14 15:11:18 -08:00
Pankaj Garg f0300c1711
Add current leader information to dashboard (#2413) 2019-01-14 14:20:05 -08:00
Rob Walker e3c0bd5a3f
remove window code from most places (#2389)
* remove window code from most places
* window used only for testing
* remove unecessary clippy directives
2019-01-14 12:11:55 -08:00
Tyera Eulberg 8af61f561b
Improve Wallet coverage (#2385)
* Add trait for RpcRequestHandler trait for RpcClient and add MockRpcClient for unit tests

* Add request_airdrop integration test

* Add timestamp_tx, witness_tx, and cancel_tx to wallet integration tests; add wallet integration tests to test-stable

* Add test cases

* Ignore plentiful sleeps in unit tests
2019-01-14 00:10:03 -07:00
Michael Vines 74e503da92 Hold an accounts_db read lock as briefly as possible to avoid deadlocking 2019-01-13 21:49:09 -08:00
Greg Fitzgerald 9328ee4f63 Revert "Revert "Delete unused code and its tests""
This reverts commit d6b3991d49.
2019-01-11 14:54:17 -07:00
Pankaj Garg d7594b19fc
Implemented a trait for vote signer service (#2386)
* Implemented a trait for vote signer service

* removes need for RPC in unit tests for vote signing

* fix build errors

* address some review comments
2019-01-11 12:58:31 -08:00
Michael Vines d6b3991d49 Revert "Delete unused code and its tests"
This reverts commit e713ba06f1.
2019-01-11 07:30:28 -08:00
Greg Fitzgerald e713ba06f1 Delete unused code and its tests 2019-01-10 23:19:38 -07:00
Greg Fitzgerald 7341298a11
Cleanup tpu forwarder (#2377)
* Use unwrap() on locks

An error there generally indicates a programmer error, not a
runtime error, so a detailed runtime message is not generally useful.

* Only clone Arcs when passing them across thread boundaries

* Cleanup TPU forwarder

By separating the query from the update, all the branches get easier to
test. Also, the update operation gets so simple, that we see it
belongs over in packet.rs.

* Thanks clippy

cute
2019-01-10 13:34:48 -07:00
Greg Fitzgerald 885fe38c01 Move BloomHashIndex into its own module
This trait is for bloom, not crds.

Slightly better would be to put it in the SDK so that the trait
implementations could go into hash and pubkey, but if we don't
want compatibility constraints, this is the next best thing.
2019-01-10 10:22:16 -08:00
Pankaj Garg 2dbe8fc1a9
Refactor vote signer code (#2368)
* Refactor vote signer code

* fixed test compilation errors

* address clippy errors

* fix missing macro_use

* move macro use

* review comments
2019-01-10 09:21:38 -08:00
Sagar Dhawan 7122139e12
Rewrite TPU forwarder test (#2344) 2019-01-10 13:50:28 +05:30
Michael Vines 4e6c03c9da Avoid holding a read lock during IO 2019-01-10 00:34:50 -07:00
Stephen Akridge 0c90e1eff6 Make entry_sender optional on window_service
window_service in replicator has no need to consume the the produced entries.
2019-01-09 15:15:47 -08:00
Stephen Akridge 491bca5e4b Remove ledger.rs
Split into entry.rs for entry-constructing functions and EntrySlice
trait and db_ledger.rs for ledger helper test functions.
2019-01-09 15:15:47 -08:00
Stephen Akridge ebd676faaa Rename Block to EntrySlice 2019-01-09 15:15:47 -08:00
Stephen Akridge 045c5e8556 Remove most of the old ledger code
Removes LedgerWriter, read_ledger, LedgerWindow
2019-01-09 15:15:47 -08:00
Stephen Akridge 45b4cf2887 Remove store_ledger_stage which is no longer needed 2019-01-09 15:15:47 -08:00
Stephen Akridge 73eca72f14 Switch test to send a repair request to try and download from replicator
Removes need for read_ledger in the test and also tests replicator
download path.
2019-01-09 13:24:12 -08:00
Rob Walker 94494b64d7 whack commented out, obsolete, superceded test 2019-01-09 11:30:07 -08:00
Rob Walker b648f37b97
encapsulate erasure_cf (#2349) 2019-01-09 10:21:55 -08:00
Greg Fitzgerald f53810fcd2 Remove unused exit variable
The exit variable was only used by a test.
2019-01-08 20:22:31 -08:00
Stephen Akridge 3c6afe7707 Rename get_blob_bytes to read_blobs_bytes 2019-01-08 16:00:39 -08:00
Stephen Akridge 09296e0d71 Fix two storage tests
* test_encrypt_files_many_keys_multiple_keys passing
  - buffer chunk size unified between single key and multiple key path,
    which shouldn't be necessary but can fix later.
* test_encrypt_file_many_keys_bad_key_length passing
2019-01-08 16:00:39 -08:00
Stephen Akridge 4b3d64ec9f Convert chacha_encrypt_file to work with db_ledger blobs directly 2019-01-08 16:00:39 -08:00
Rob Walker a904e15ecc
enscapsulate data_cf (#2336)
* enscapsulate data_cf
2019-01-08 15:53:44 -08:00
Greg Fitzgerald a82a5ae184 Delete unused code
The ignored test is still broken, but at least no longer creates a
window for no reason.

Also removed all remaining references to "ncp".
2019-01-08 14:09:50 -08:00
Rob Walker bafd90807d
encapsulate meta_cf (#2335) 2019-01-08 11:41:55 -08:00
jackcmay 0f8ea6872e
Add missing error counters and load_account test cases (#2327) 2019-01-08 09:20:25 -08:00
Greg Fitzgerald d2431128c7 Remove WriteStage from TPU/TVU diagrams
Fixes #2312
2019-01-08 08:42:06 -08:00
Michael Vines a8b9899dee Add retry, restore ignored tests 2019-01-07 19:30:08 -08:00
Stephen Akridge d2cb4e003c Re-enable the --lib tests 2019-01-07 15:28:20 -08:00
Greg Fitzgerald 6000df9779 Optimize has_duplicates() for short slices 2019-01-07 13:20:04 -07:00
Greg Fitzgerald 24963e547c with_subset() -> get_subset_unchecked_mut()
A simpler, safer, and better documented use of unsafe code
2019-01-07 13:20:04 -07:00
Michael Vines 46d44ca99c Add make_rpc_request retry mechanism 2019-01-07 11:02:35 -08:00
jackcmay 0c52df7569
Consolidate locks and error handling when loading accounts(#2309) 2019-01-06 22:06:55 -08:00
Pankaj Garg 91bd38504e
Use vote signer service in fullnode (#2009)
* Use vote signer service in fullnode

* Use native types for signature and pubkey, and address other review comments

* Start local vote signer if a remote service address is not provided

* Rebased to master

* Fixes after rebase
2019-01-05 12:57:52 -08:00
Michael Vines b7dc9dbc76 RPC API now assumes a drone running on the bootstrap leader 2019-01-04 18:45:55 -08:00
Michael Vines 8b357dcb32
cargo fmt 2019-01-04 16:39:04 -08:00
Michael Vines 1f6346d880 De-dup ledgers - db_ledger is now the only ledger written to disk 2019-01-04 16:37:00 -08:00
Jack May b7bd38744c Spelling and formatting 2019-01-04 16:04:31 -08:00
Jack May f8a67e282a Ignore test_tpu_forwarder (#2307) 2019-01-04 16:02:50 -08:00
Greg Fitzgerald 0505d7bd32 Don't double-clone every account 2019-01-03 17:42:37 -07:00
Pankaj Garg 00d310f86d
Remove some metrics datapoint, as it was causing excessive logging (#2287)
- 100 nodes test was bringing down the influx DB server
2019-01-03 09:25:11 -08:00
Sagar Dhawan 0bea870b22
Dynamic N layer 'avalanche' broadcast and retransmit (#2058)
* Dynamic N layer avalanche broadcast and retransmit
2019-01-02 14:16:15 +05:30
Michael Vines 58f2598d5d Revert "Validators make a transaction to advertise their storage last_id"
This reverts commit a1759aed19.
2018-12-23 14:02:09 -08:00
carllin 58a4905916
Make reconstruct_entries_from_blobs() support Blobs and borrowed SharedBlobs, make distinction between to_blobs and to_shared_blobs (#2270) 2018-12-22 19:30:30 -08:00
Michael Vines 2c9607d5da Rename getConfirmation -> getConfirmationTime 2018-12-22 12:47:02 -08:00
Stephen Akridge a1759aed19 Validators make a transaction to advertise their storage last_id
* Also implement more storage contract logic
* Add transactions for proof validation,
* Move storage state members into system storage account userdata
2018-12-21 15:45:30 -08:00
Sathish 1a3387706d
Spawn threads based on cpu count (#2232) 2018-12-21 13:55:45 -08:00
Pankaj Garg 951d6398a0
Rename finality to confirmation (#2250)
* Rename finality to confirmation

* fix cargo fmt errors
2018-12-20 15:47:48 -08:00
Pankaj Garg 7c98545b33
Use newer votes to calculate confirmation time (#2247) 2018-12-20 15:27:47 -08:00
Pankaj Garg bb1060bdad
Reduce ticks per block to increase voting frequency (#2242) 2018-12-20 14:43:03 -08:00
Michael Vines 7ad45a91ec Fix compile error 2018-12-20 13:47:36 -08:00
Michael Vines 034c5d0422 db_ledger now fully encapsulates rocksdb 2018-12-20 12:32:25 -08:00
carllin 7148c14178
Debug broadcast (#2233)
* Account for duplicate blobs in process_blobs

* Increase max bytes for level base to match write buffer
2018-12-20 12:12:04 -08:00
Michael Vines 93fb61dc8f Re-export rocksdb::DBRawIterator until it can be encapsulated 2018-12-20 10:38:03 -08:00
Michael Vines b36ceb5be4 Remove rocksdb dependency from result.rs 2018-12-20 10:38:03 -08:00
Michael Vines 37d7ad819b Purge DB::destroy() usage 2018-12-20 10:38:03 -08:00
carllin 666af1e62d
Debug broadcast (#2208)
* Add per cf rocksdb options, increase compaction and flush threads

* Change broadcast stage to bulk write blobs

* add db_ledger function specifically for broadcast

* fix broken tests

* fix benches
2018-12-19 16:11:47 -08:00
Rob Walker b4f4347d6e
add some more tests (#2217) 2018-12-18 17:27:03 -08:00
Rob Walker 5c866dd000
test drive new coverage stuff (#2216) 2018-12-18 16:44:27 -08:00
Pankaj Garg 974249f2a5
Parallelize entry processing in replay stage in validators (#2212)
* Parallelize entry processing in replay stage in validators

- single threaded entry processing is not utlizing CPU cores to the fullest

* fix tests and address review comments
2018-12-18 16:06:05 -08:00
Rob Walker a65022aed7
DbLedger doesn't need to be mut, doesn't need an RwLock (#2215)
* DbLedger doesn't need to be mut, doesn't need an RwLock

* fix erasure cases
2018-12-18 15:18:57 -08:00
Sagar Dhawan ffdc1814c6 Add counters for gossip verification failures (#2094) 2018-12-17 20:12:50 -08:00
carllin 9720ac0019
Fix try_erasure() (#2185)
* Fix try_erasure bug

* Re-enable asserts in test_replicator_startup

* Add test for out of order process_blobs
2018-12-17 15:34:19 -08:00
anatoly yakovenko 9a6e27ac36
Accounts is to big, should be its own module (#2198)
Account module is to big, should be in its own module.
2018-12-17 12:41:23 -08:00
Michael Vines 07202205c4 Revert "ignore unstable tests"
This reverts commit bd7ef5d445071329a3b49b1f8be71b602226bbec.
2018-12-17 10:47:32 -08:00
Michael Vines b7144560c9 Include port number when gossip bind_to fails 2018-12-17 09:31:31 -08:00
anatoly yakovenko 4be6d01dfb
Move last ids (#2187)
* Break out last_ids into its own module
* Boot SignatureNotFound from BankError
* No longer return BankError from LastIds methods
* No longer piggypack on BankError for a LastIds signature status
* Drop all dependencies on the bank
* SignatureStatus -> Status and LastIds -> StatusDeque
* Unstable tests, issue 2193
2018-12-17 07:55:56 -08:00
Michael Vines fd562cb9e2 Rust 2018 cleanup 2018-12-14 21:57:15 -08:00
Michael Vines 5bf9a20d42 fullnode-config no longer depends on src/ 2018-12-14 20:13:34 -08:00
Michael Vines eedc8c7812 Move src/netutil.rs into its own crate 2018-12-14 20:13:34 -08:00
carllin 9ef5e51c0f
Cleanup slot remnants in db_ledger (#2153)
* Cleanup slot remnants in db_ledger
2018-12-14 17:05:41 -08:00
Sathish 8ee0e9632c
Switch to using hashbrown version of HashMap and (#2158)
HashSet for improved performance and memory usage
2018-12-14 15:10:10 -08:00
Michael Vines 6ac466c0a4 Move src/logger.rs into logger/ crate to unify logging across the workspace 2018-12-14 13:10:43 -08:00
Michael Vines d45fcc4381 Move src/wallet.rs into wallet/ crate 2018-12-14 12:15:18 -08:00
carllin 6fcccedb70
align tick entries' tick_height with actual number of ticks in bank (#2147) 2018-12-14 02:25:50 -08:00
Michael Vines 0fe6d61036 Move binaries from src/bin into their own crate 2018-12-13 22:25:27 -08:00
Pankaj Garg 091b21fae7
Vote every number of ticks (#2141)
* Vote every number of ticks

* address review comments

* fix for failing leader rotation tests

* remove check for vote failure from replay tests
(as votes will be cached and transmitted when leader is available)
2018-12-13 18:43:10 -08:00
Sathish 7325b19aef
Do not allocate for each metrics submission (#2146) 2018-12-13 16:40:00 -08:00
Stephen Akridge 7cdbbfa88e Storage stage updates
* Remove logging init from storage program: saw a crash in a test
  indicating the logger being init'ed twice.
* Add entry_height mining proof to indicate which segment the result is
  for
* Add an interface to get storage miner pubkeys for a given entry_height
* Add an interface to get the current storage mining entry_height
* Set the tvu socket to 0.0.0.0:0 in replicator to stop getting entries
  after the desired ledger segment is downloaded.
* Use signature of PoH height to determine which block to download for
  replicator.
2018-12-13 11:30:12 -08:00
Rob Walker 9880a86f80
remove prev_id, unused (#2150) 2018-12-13 09:24:38 -08:00
Michael Vines 647e5d76b0 Move solana-fullnode into fullnode/ 2018-12-13 01:45:29 -08:00
Michael Vines 7e4af9382e Move solana-upload-perf into upload-perf/ 2018-12-13 01:06:40 -08:00
Michael Vines 282d4a3563 Move solana-keygen into keygen/ 2018-12-13 01:06:40 -08:00
Michael Vines cafeef33c3 Relocate all keypair generation into one location: sdk/src/signature.rs 2018-12-13 01:06:40 -08:00
Rob Walker 4f48f1a850
add db_ledger genesis, rework to_blob(), to_blobs() (#2135) 2018-12-12 20:42:12 -08:00
Carl a05a378db4 cleanup 2018-12-12 19:12:51 -08:00
Carl 245362db96 Make a dummy version of serving repairs from db_ledger 2018-12-12 19:12:51 -08:00
Mark b1b190b80d Fix too many args in Tvu::new (#2114)
* Reduce args in Tvu::new under to 8

Now pass in sockets through a the crate::tvu::Sockets struct

Move ClusterInfo.keypair to pub(crate) in order to remove redundant
signing keypair parameter

* remove commented code
2018-12-12 18:57:48 -08:00
Rob Walker 3408ce89a7
add check_tick_height (#2144) 2018-12-12 18:52:11 -08:00
Michael Vines 75f407e191 Provide entire elf to bpf_loader 2018-12-12 17:14:50 -08:00
Mekagoza 9b81696a09 remove obsoleted TODO 2018-12-12 16:26:59 -08:00
Mekagoza 80e19e0ad7 Encapsulate accounts of solana:🏦:Accounts
Make the field private and expose an account_values() method that
returns the values iterator from the internal hashmap
2018-12-12 16:26:59 -08:00
carllin ae903f190e
Broadcast for slots (#2081)
* Insert blobs into db_ledger in broadcast stage to support leader to validator transitions

* Add transmitting real slots to broadcast stage

* Handle real slots instead of default slots in window

* Switch to dummy repair on slots and modify erasure to support leader rotation

* Shorten length of holding locks

* Remove logger from replicator test
2018-12-12 15:58:29 -08:00
Pankaj Garg 3413ecc2bd
Change query used to find list of nodes in the network (#2124)
* Change query used to find list of nodes in the network

* include "All" option for host selection
2018-12-12 12:38:00 -08:00
Sagar Dhawan 4788a4f775
Correctly describe repair and retransmit peers (#2110) 2018-12-11 15:51:47 -08:00
Pankaj Garg 9243bc58db
Metrics for window repair (#2106)
* Metrics for window repair

- Also increase max repair length

* fix vote counters, and add repair window graph

* update per node graphs

* revert max repair length change
2018-12-11 15:43:41 -08:00
Rob Walker 4d67aca919
add genesis and read_ledger to db_ledger (#2097) 2018-12-11 09:14:23 -08:00
Rob Walker aeee25e703
add tick_height to Entry to be able to repair by period, chain forks of Entries, etc. (#2096) 2018-12-10 20:03:04 -08:00
Greg Fitzgerald 01c524ddd2 Revert changes to counter names 2018-12-10 15:26:43 -07:00
Greg Fitzgerald 5e703dc70a Free up the term 'replicate' for exclusive use in replicator
Also, align Sockets field names with ContactInfo.
2018-12-10 15:26:43 -07:00
Pankaj Garg bc96bd3410
Fix peer count in edge dashboard (#2090)
Fixes #2075
2018-12-10 14:24:32 -08:00
Michael Vines 4ea1c030bc Give bootstrap leader one more token 2018-12-10 13:24:14 -08:00
Stephen Akridge 172e511e56 Use retry_transfer to test multiple times for replicator tokens
May fix failures in CI where replicator is trying to do an airdrop.
2018-12-10 12:19:00 -08:00
Tyera Eulberg 4481efd51e
Merge pull request #2084 from CriesofCarrots/fix-wallet-accountinuse
Fix wallet accountinuse
2018-12-10 12:20:55 -07:00
Greg Fitzgerald 0ef099421c cargo fmt 2018-12-08 23:19:55 -07:00
Greg Fitzgerald f1ae5b1795 Fix warnings 2018-12-08 23:19:55 -07:00
Greg Fitzgerald a8d6c75a24 cargo +nightly fix --features=bpf_c,cuda,erasure,chacha --edition-idioms 2018-12-08 23:19:55 -07:00
Greg Fitzgerald c49e2f8bbd cargo +nightly fix --features=bpf_c,cuda,erasure,chacha --edition 2018-12-08 23:19:55 -07:00
Greg Fitzgerald ec5a8141eb cargo fix --edition 2018-12-08 23:19:55 -07:00
Sathish 2de45a4da5
Update airdrop tokens to 3 for fullnode (#2051)
Filter out leader while computing the super majority stake
2018-12-08 16:54:42 -08:00
Michael Vines 73b9ee9e84 Add solana_ prefix to native_loader program
This allows its logging to show up in the default RUST_LOG=solana=info
log setting
2018-12-08 11:04:45 -08:00
Greg Fitzgerald 0a83b17cdd
Upgrade to Rust 1.31.0 (#2052)
* Upgrade to Rust 1.31.0
* Upgrade nightly
* Fix all clippy warnings
* Revert relaxed version check and update
2018-12-07 20:01:28 -07:00
Tyera Eulberg 571522e738 Update jsonrpc version 2018-12-07 17:47:54 -07:00
Stephen Akridge 3441d3399b Replicator rework
* Move more of the replicator logic into the replicator class
* Add support for the RPC interface to query the storage last_id value
  that the replicator would sign and use to pick a block.
* Fix replicator connecting to gossip and change test to exercise that
  scenario.
2018-12-07 15:20:36 -08:00
Michael Vines f4b26247c0 Genesis only needs a keypair, not the entire fullnode::Config 2018-12-06 16:31:24 -08:00
Michael Vines c4b8f0cd2f bench-tps will now generate an ephemeral identity if not provided with one
Also simplify scripts as a result
2018-12-06 16:30:48 -08:00
Greg Fitzgerald 97b1156a7a Rename Ncp to GossipService
And BroadcastStage to BroadcastService since it's not included in the
TPU pipeline.
2018-12-06 15:48:19 -07:00
Michael Vines cc2f448d92 Add fullnode --no-leader-rotation flag 2018-12-06 11:30:19 -08:00
Michael Vines 632425c7d7 Move native_loader under programs/native/ 2018-12-05 14:32:42 -08:00
dependabot[bot] ad3e36a7ab Bump rand from 0.5.5 to 0.6.1 (#1891)
* Bump rand from 0.5.5 to 0.6.1

Bumps [rand](https://github.com/rust-random/rand) from 0.5.5 to 0.6.1.
- [Release notes](https://github.com/rust-random/rand/releases)
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/rand/commits)

Signed-off-by: dependabot[bot] <support@dependabot.com>

* Fix conflicts and deprecated usages

* Fix benches
2018-12-05 14:12:10 -08:00
Michael Vines 1bcafca690
Find test_tx again 2018-12-05 13:29:29 -08:00
Rob Walker e21b6d9db3
ensure we'd actually have N hashes per tick (#2011) 2018-12-05 12:49:41 -08:00
carllin 9c30bddb88
Rocks db erasure decoding (#1900)
* Change erasure to consume new RocksDb window

* Change tests for erasure

* Remove erasure from window

* Integrate erasure decoding back into window

* Remove corrupted blobs from ledger

* Replace Erasure result with result module's Result
2018-12-05 12:47:19 -08:00
Michael Vines 59e6bd115e system_program must be a static lib as it allocates Account memory 2018-12-05 10:49:06 -08:00
Michael Vines 15aef079e3 Include builtin programs for ledger verification 2018-12-05 10:49:06 -08:00
Michael Vines 42689d4842 cargo fmt 2018-12-05 10:49:06 -08:00
Michael Vines 214ed3667c Move system_transaction out of src/ 2018-12-05 10:49:06 -08:00
Michael Vines 122627dda2 Move loader_transaction out of src/ 2018-12-05 10:49:06 -08:00
Michael Vines 7af95eadcc Move vote_transaction out of src/ 2018-12-05 10:49:06 -08:00
Michael Vines 9ee858a00c Move budget_program out of src/ 2018-12-05 10:49:06 -08:00
Michael Vines 27d456bf93 Move storage_program out of src/ 2018-12-05 10:49:06 -08:00
Michael Vines ea6e042a6f Move vote_program out of src/ 2018-12-05 10:49:06 -08:00
Michael Vines a594f56c02 Add token_program.rs to sdk/ 2018-12-05 10:49:06 -08:00
Michael Vines e6fa74fe69 Remove custom Error enum, just use ProgramError 2018-12-05 10:49:06 -08:00
Michael Vines 9a4f8199d6 Move system_program out of src/ 2018-12-05 10:49:06 -08:00
Michael Vines ae0be1e857 Remove bpf_loader.rs 2018-12-05 10:49:06 -08:00
Michael Vines d010cac8a5 Remove token_program.rs 2018-12-05 10:49:06 -08:00
Michael Vines 63a758508a Add sdk native_loader.rs 2018-12-05 10:49:06 -08:00
Stephen Akridge c1c2f1f0a9 Cleanup ad-hoc rpc address formation
Lots of places where we are forming rpc addresses.
2018-12-03 18:13:55 -08:00
Michael Vines 777a0a858e Move ProgramError into sdk/ 2018-12-03 13:50:00 -08:00
Michael Vines 68e99c18c0 Remove duplicate SYSTEM_PROGRAM_ID 2018-12-03 13:50:00 -08:00
Michael Vines c99f93e40a Remove signature.rs indirection 2018-12-03 13:50:00 -08:00
carllin 4ae58cc854
Change range of leader scheduler to match current broadcasts (#1920) 2018-12-03 00:10:43 -08:00
Sagar Dhawan b21facab7b
Add metrics for prune messages (#1981) 2018-12-01 14:05:40 -08:00
Sagar Dhawan 34c3a0cc1f
Add signature verification to gossip (#1937) 2018-12-01 12:00:30 -08:00
Pankaj Garg 549bfe7412
Vote signing JSON RPC service (#1965)
* Vote signing JSON RPC service

- barebone service that listens for RPC requests

* Daemon for vote signer service

* Add request APIs for JSON RPC

* Cleanup of cargo dependencies

* Fix compiler error
2018-11-30 15:07:08 -08:00
Rob Walker 3ca826a480
re-enable test_tpu_forwarder (#1964) 2018-11-30 13:52:37 -08:00
Michael Vines bad0b55ab6 Expose which keys signed the Transaction in the SDK 2018-11-30 08:16:23 -08:00
Michael Vines 0878bd53d9 Delete stub src/transaction.rs 2018-11-29 23:07:57 -08:00
Stephen Akridge de910e1169 Make test_pubkey_distribution faster
multi-thread pubkey histogram generation.
2018-11-29 17:37:37 -08:00
Stephen Akridge 9684737de7 Add wait before checking confirm again
Otherwise we can quickly check that we
have no signature 4 times in a row.
2018-11-29 15:32:58 -08:00
Stephen Akridge ecc87ab1aa Add a an optional timeout to thin_client
Such that a negative test like test_transaction_count doesn't
have to wait num_retries * default_timeout.
2018-11-29 13:53:40 -08:00
Michael Vines fa359c6fc4 Merge vote new and register transactions 2018-11-29 12:31:34 -08:00
Stephen Akridge 8cc751d1cc Improve RPC service startup error messages with actual error
Error always fixed to message about ports but that's not the only
error that can occur.
2018-11-29 11:39:42 -08:00
Stephen Akridge 978fd6858f Move replicator_startup_test to integration test set
Sometimes fails when run multithreaded with other tests.
2018-11-29 11:39:42 -08:00
Michael Vines 41689256c6 Ensure key[0] is signed 2018-11-29 10:26:46 -08:00
Pankaj Garg 194e3100a9
Additional checks in test_bank_checkpoint_zero_balance (#1943) 2018-11-28 12:40:34 -08:00
Rob Walker 8ad4464d4b add tests for other "from" indexes signing (or not) 2018-11-28 07:56:04 -08:00
Rob Walker e7b0a736f5 verify signature is on the from account 2018-11-28 07:56:04 -08:00
Rob Walker 167eb01735 optimize bench-tps and rpc_request to work on crappy WSL boxes 2018-11-27 22:45:08 -08:00
Michael Vines 8fb5d72b13 Make insufficient tokens message more helpful 2018-11-27 17:37:25 -08:00
Pankaj Garg 8947c5a4aa
Set account to default if the balance reaches 0 in a checkpoint bank (#1932)
Fixes: #1931
2018-11-27 14:17:29 -08:00
Greg Fitzgerald a7562c9be1 Extract execute_transaction() from the bank 2018-11-27 12:35:52 -07:00
Greg Fitzgerald 08dc169f94 Hoist load_loaders()
This makes execute_transactions() stateless.
2018-11-27 12:35:52 -07:00
Greg Fitzgerald f549d8ac74 Hoist loading of loaders
This might cause a TPS boost in batched BPF transactions, since
now it'll only clone its account once per transaction instead of
once per instruction.
2018-11-27 12:35:52 -07:00
Greg Fitzgerald 1ac7536286 Pass executable_accounts into with_subset() 2018-11-27 12:35:52 -07:00
Greg Fitzgerald 88e270723f Move markdown book out of src/ 2018-11-26 09:44:19 -08:00
Michael Vines a13e25f083 Ignore flaky test_tpu_forwarder 2018-11-26 09:27:21 -08:00
Greg Fitzgerald 3d3a30e200 Fix mdbook test 2018-11-26 07:51:10 -08:00
Greg Fitzgerald 76b83ac0f4 Move testnet demos into the book
Have git readme focus on fullnode development and the book focus on
users.
2018-11-26 07:51:10 -08:00
Greg Fitzgerald 903a9bfd05 s/contract/program/ 2018-11-26 08:20:42 -07:00
Greg Fitzgerald 655ee1a64b Fix typos 2018-11-26 08:20:42 -07:00
Greg Fitzgerald e0e6c3fdb2 Extract execute_instruction() to seed new runtime module
Fixes #1528
2018-11-26 08:20:42 -07:00
Greg Fitzgerald 31f00974f2 Hoist the lookup of executable accounts 2018-11-26 08:20:42 -07:00
Greg Fitzgerald c3218bb9c2 Hoist tick_height 2018-11-26 08:20:42 -07:00
Greg Fitzgerald d2972024de Uppercase acronyms
Looks like there will be very little Rust code in the markdown book
so switching back to English capitalization conventions.
2018-11-25 22:58:07 -07:00
Greg Fitzgerald dc0a2ca656
Move disclaimer down a bit
Odd to see a disclaimer before knowing anything about what you're reading
2018-11-25 20:27:35 -07:00
Greg Fitzgerald c3fb9d5549
Cleanup book (#1904)
* Cleanup book

* Distinguish upstream from downstream validators
* Add BroadcastStage to Fullnode/Tpu diagrams
* First attempt to re-describe the runtime

* Reorg book

Push back details of the fullnode implementation
2018-11-25 16:58:38 -07:00
Sathish f5b5c54d7d
Update condition for nosigverify (#1903) 2018-11-25 13:11:07 -08:00
Sathish 9f0b06bb86 Filter out leader node while retransmitting blobs (#1894) 2018-11-24 20:33:49 -08:00
carllin 57a384d6a0
Rocks db window service (#1888)
* Add db_window module for windowing functions from RocksDb

* Replace window with db_window functions in window_service

* Fix tests

* Make note of change in db_window

* Create RocksDb ledger in bin/fullnode

* Make db_ledger functions generic

* Add db_ledger to bin/replicator
2018-11-24 19:32:33 -08:00
Greg Fitzgerald 69802e141f Add the story of how this codebase came to be 2018-11-24 14:39:53 -07:00
Greg Fitzgerald 6fc02b7424 Detect legacy programs upfront 2018-11-24 11:56:51 -07:00
Greg Fitzgerald 30cdd85028 Implement the same interface in all builtin programs 2018-11-24 11:56:51 -07:00
Greg Fitzgerald 871dd47019 Extract the part of execute_instruction that should only return a ProgramError
TODO: hoist load_executable_accounts() and then change
process_instruction() to return ProgramError.
2018-11-24 11:56:51 -07:00
Greg Fitzgerald 37f8dd57e2 Extract ProgramError from BankError 2018-11-24 11:56:51 -07:00
Greg Fitzgerald f827bfd83f Remove instruction index parameter 2018-11-24 11:56:51 -07:00
Greg Fitzgerald b3af930153 Rename process_transaction to process_instruction 2018-11-24 11:56:51 -07:00
Greg Fitzgerald cd488b7d07 Hoist program static methods to top-level functions 2018-11-24 11:56:51 -07:00
Rob Walker e2373ff51a add nosigverify command line option to ease debug 2018-11-23 16:55:04 -08:00
Greg Fitzgerald b3d2c900cd Rename BudgetState to BudgetProgram 2018-11-23 13:25:17 -07:00
Michael Vines d5adec20a3 get_ip_addr: Fall back to loopback if no better option exists 2018-11-23 13:24:41 -05:00
carllin 942256a647
Add db_ledger benchmarks (#1875)
* Add db_ledger benchmarks

* ignore benches in CI, due to timeouts
2018-11-23 06:12:43 -08:00
carllin a3321a5d80
Fix endianess in db_ledger to account for the default byte-comparator used by Rocksdb (#1885) 2018-11-22 01:35:19 -08:00
carllin 521de13571
Add maximum repair length to db_window (#1886)
* Add maximum repair length to db_window
2018-11-21 23:44:49 -08:00
Greg Fitzgerald 3abf6a8a30 Reorg the markdown book to cater to app devs
First, talk about how a client interacts with Solana to do useful
things. Then describe how the fullnode you're talking to works and
why it's so very fast.  Last, why that fullnode you don't trust
does what you asked it to anyway.
2018-11-21 15:49:57 -08:00
jackcmay 59163e2dd9
Optimize some CI stuff (#1880)
* CI Optimizations
2018-11-21 12:16:16 -08:00
carllin d9dabdfc74
Rocks db window utils (#1851)
* Implement new ledger module based on RocksDb

* Add db_window module for windowing functions from RocksDb
2018-11-19 23:20:18 -08:00
Rob Walker 6b910d1bd4 add tpu_forwarding, simplify ClusterInfo::new() from Result<Self> to Self 2018-11-19 20:45:49 -08:00
Rob Walker bbd9ea8c00
Delete settings.rs.foo 2018-11-19 13:39:08 -08:00
Anatoly Yakovenko fc67a968e8 Use known keys in the unit test to avoid random false positives. 2018-11-19 13:41:24 -07:00
Rob Walker 3d113611cc
remove Result<> return from ClusterInfo::new() (#1869)
strip Result<> for ClusterInfo::new()
2018-11-19 11:25:14 -08:00
Sagar Dhawan 07667771ef
Fix Gossip Pushes going to invalid addresses (#1858) 2018-11-17 19:57:28 -08:00
Michael Vines 3822c29415 Route program_id to program entrypoint 2018-11-17 19:42:03 -08:00
Greg Fitzgerald ff386d6585 Add disclaimer to markdown book
copy-paste from readme
2018-11-17 19:56:08 -07:00
Greg Fitzgerald 88ddb31477 teminology cleanup: leader slots and voting rounds 2018-11-17 18:56:13 -07:00
Greg Fitzgerald 077d1a41f1 Add too book 2018-11-17 18:56:13 -07:00
Greg Fitzgerald 857ab8662e backticks and missing variable descriptions 2018-11-17 18:56:13 -07:00
Greg Fitzgerald a17f9bd0f4 Work towards adding leader rotation to the book 2018-11-17 18:56:13 -07:00
Greg Fitzgerald f4b9e93b11 Migrate storage RFC to book 2018-11-17 18:55:08 -07:00
Greg Fitzgerald 2c11bf2e66 Various book cleanup
* Merge Leader and Validator diagrams
* New sdk-tools diagram
* Move terminology to just after introduction
* Purge use of LAMPORT as an acronym
* Add notes about persistent storage
2018-11-17 17:50:29 -08:00
Michael Vines 719e14b30a Add an explicit state of a reserved signature
An RPC client that fetches the signature status before the bank finishes
executing the corresponding Transaction should receive SignatureNotFound
instead of Confirmed
2018-11-17 16:40:23 -08:00
Michael Vines d1f01b5209 Fix clippy lint 2018-11-17 15:54:21 -08:00
Michael Vines d96a6b42a5 Move drone into its own crate 2018-11-16 20:42:21 -08:00
carllin cf95708c18 Set drone address to always be the initial network entry point (#1847)
* Set drone address to always be the initial network entry point, so that even when leaders rotate the client can still find the drone

* Extract drone address as a separate argument to bench-tps

* Add drone port to client.sh instead of setting it in bench-tps

* Add drone entrypoint to scripts

* Fix build error
2018-11-16 19:56:26 -08:00
Michael Vines e1c7b99450
Accounts get kicked if no tokens 2018-11-16 18:53:37 -08:00
Sagar Dhawan 12ae7b9a6b
Add test for tvu POH verification (#1844) 2018-11-16 15:48:10 -08:00
Michael Vines 6ac5700f2e Move metrics into its own crate 2018-11-16 15:10:07 -08:00
Michael Vines a0dd8617be Remove airdrop from fullnode 2018-11-16 13:25:55 -08:00
Rob Walker 1576072edb
remove spurious eprintln!() 2018-11-16 10:21:58 -08:00
carllin 03d206a7ca
Check for valid tvu, not tpu in broadcast (#1836) 2018-11-15 23:30:22 -08:00
Sathish c973de1d76
Decouple log and metrics rate (#1839)
Use separate env for log and metrics rate.

Set default log level to WARN if unset.
2018-11-15 22:27:16 -08:00
Stephen Akridge 71336965a6 Limit targets to 4 in bench-tps
Transaction got bigger so can only fit 4 targets in a
Transaction now.
2018-11-15 20:25:07 -08:00
Michael Vines e791d0f74d Drone now returns signed airdrop transactions 2018-11-15 17:13:13 -08:00
Stephen Akridge 3543a9a49f Add check for missing signature with fee'ed transaction
And update fetch-perf-libs version
2018-11-15 16:23:13 -08:00
Stephen Akridge 7dd198a99e Change signed_key to index into account_keys
If index is within the signed keys range.
2018-11-15 16:23:13 -08:00
Stephen Akridge e048116ab2 Remove signed_keys
Use first signatures.len() of account_keys for signing
2018-11-15 16:23:13 -08:00
Stephen Akridge cda9ad8565 Multiple signatures for transactions
With multiple instructions in a TX may need
multiple signatures.

Fixes #1531
2018-11-15 16:23:13 -08:00
carllin 928f375683
Rocks db (#1792)
* Add rocksdb crate

* Implement new ledger module based on RocksDb
2018-11-15 15:53:31 -08:00
Rob Walker d3e521f70e
accept other socket errors, ignore unless out of tries (#1835) 2018-11-15 15:49:37 -08:00
Michael Vines a7ee428214 Fix build 2018-11-15 14:06:57 -08:00
anatoly yakovenko a41254e18c
Add scalable gossip library (#1546)
* Cluster Replicated Data Store

Separate the data storage and merge strategy from the network IO boundary.
Implement an eager push overlay for transporting recent messages.

Simulation shows fast convergence with 20k nodes.
2018-11-15 13:23:26 -08:00
Sagar Dhawan c81a3f6ced
Fix RPC address clashes on local multi-node testnet (#1821)
* Fix RPC address clashes on local multi-node testnet
2018-11-15 10:42:02 -08:00
Tyera Eulberg a5412fc0cd Fix find port functions 2018-11-15 10:45:39 -07:00
Michael Vines 5a8938209b Expose tick_height to native programs 2018-11-14 10:33:27 -08:00
Michael Vines e33f3a2562 Publish expected native program entrypoint in sdk/ 2018-11-14 10:33:27 -08:00
Sathish 9fe210c454
Add host information to db entries (#1778)
Add new field to each db entry identifying the host
that it originated from.
2018-11-13 21:54:15 -08:00
Jack May e7b6c8b7e0 Accounts get kicked if no tokens 2018-11-13 17:23:13 -08:00
Jack May 4e553ea095 test_replicate fails locally, ignore 2018-11-13 17:13:25 -08:00
Jack May 0c46f15f94 test_rpc_new fails locally, ignore for now 2018-11-13 17:12:25 -08:00
Sathish 7b92497d21
Update counters irrespective of logging level (#1799) 2018-11-13 16:55:14 -08:00
Sagar Dhawan 729d28d910 Add poh verification before processing entries
- Replicate stage now verifies entries delivered
  by the window
- Minor refactor of entries_from_blobs
2018-11-13 14:17:00 -08:00
carllin 6335be803c
Broadcast last tick before leader rotation (#1766)
* Broadcast last tick before leader rotation to everybody on network

* Add test

* Refactor broadcast
2018-11-13 02:21:37 -08:00
Sagar Dhawan a77b1ff767
Revert "Migrate from ring to ed25519-dalek" (#1798)
* Revert "Migrate from ring to ed25519-dalek"

This reverts commit 7c610b216b.

* Fix test failures with revert
2018-11-12 22:34:43 -08:00
Tyera Eulberg c7a67b5a02 Add deploy command to test 2018-11-12 18:21:16 -07:00
Tyera Eulberg 0e749dad4c Use cluster_info to get rpc address 2018-11-12 18:21:16 -07:00
Rob Walker fa72160c95
add last_id to Entry, PohEntry (#1783)
add prev_id to Entry, PohEntry
2018-11-12 17:03:23 -08:00
Greg Fitzgerald 126f065cc9 Extract complex loop from execute_instruction 2018-11-12 14:47:23 -08:00
carllin 4d98da44e3
Fix possibility of a vote error breaking ledger (#1768)
* Fix possibility of a vote error breaking ledger

* Add test
2018-11-12 11:40:32 -08:00
Michael Vines 522876c808 Rename Account.program_id to Account.owner 2018-11-12 10:59:01 -08:00
Michael Vines 7d05cc8c5d Add missing account fields 2018-11-12 10:59:01 -08:00
Michael Vines 49f4be6a2b codemod --extensions rs loader_program_id loader 2018-11-12 10:59:01 -08:00
Greg Fitzgerald 5fce8d2ce1 Don't ignore VoteProgram errors 2018-11-11 22:18:06 -07:00
Greg Fitzgerald 2696b22348 Cleanup TVU diagram 2018-11-11 20:55:21 -08:00
Greg Fitzgerald 5df4754579 Don't call instructions transactions 2018-11-11 20:07:15 -08:00
Tyera Eulberg a00284c727 Remove userdata diff and make helper fn 2018-11-11 18:57:28 -07:00
Tyera Eulberg 3832602ec4 Move notifications after store_accounts 2018-11-11 18:57:28 -07:00
Greg Fitzgerald fea86b2955 No longer serialize as JSON-encoded pkcs8
That's supposed to be an ASCII format, but we're not making use
of it. We can switch back to that some day, but if we do, it shouldn't
be JSON-encoded.
2018-11-11 09:22:52 -07:00
Greg Fitzgerald 7c610b216b Migrate from ring to ed25519-dalek
Why?

* Pure Rust, no BoringSSL (or OpenSSL) dependency
* Those avx2 benchmarks
* ring includes far more than what we need
* ring author won't add release tags: https://github.com/briansmith/ring#versioning--stability
2018-11-11 09:22:52 -07:00
Greg Fitzgerald cdf1a96e23 Revert "V1 Window/Ledger based on RocksDb (#1712)"
This reverts commit bfcdec95cb.
2018-11-09 20:25:53 -07:00
carllin bfcdec95cb
V1 Window/Ledger based on RocksDb (#1712)
* Add rocksdb

* Implement new ledger module based on RocksDb
2018-11-09 18:30:26 -08:00
Greg Fitzgerald fc55835932 Revert "Boot rpc_port"
This reverts commit 1984b6db06dae4f6b655b9076e657e389d786105.
2018-11-09 17:52:10 -07:00
Greg Fitzgerald 3772910bf2 Boot rpc_port 2018-11-09 17:52:10 -07:00
Greg Fitzgerald 24379c14dc Fix clippy warnings 2018-11-09 17:52:10 -07:00
Greg Fitzgerald 23846bcf1c Don't require a cluster to query for one's own pubkey 2018-11-09 17:52:10 -07:00
Greg Fitzgerald 9dd0a6e6a7 Boot drone_addr and rpc_addr from config
WalletConfig is intended for the validated command-line input.
2018-11-09 17:52:10 -07:00
Greg Fitzgerald 5ca473ac2d Don't get the network from parse_args 2018-11-09 17:52:10 -07:00
Greg Fitzgerald f5827d4a83 Fix typo 2018-11-08 17:15:48 -07:00
Greg Fitzgerald b0f8a983c4
Add the solana-wallet documentation (#1744)
* Add the solana-wallet documentation

There doesn't seem to be a way to publish bin docs to crates.io.
Until there is, we can include CLI documentation is the appendix
of the markdown book.

* A command to generate all the usage docs

Usage:

$ scripts/wallet-help.sh >> src/wallet.md
2018-11-08 15:42:20 -07:00
Rob Walker d831c5dcc9
remove dead poh code (#1747) 2018-11-08 12:55:23 -08:00
Greg Fitzgerald ce474eaf54 Better titles for tpu and tvu 2018-11-08 11:33:52 -07:00
Greg Fitzgerald 0da1c06b15 Add disk to the hardware used by both Tpu and Tvu 2018-11-08 11:33:52 -07:00
Greg Fitzgerald 01edc94a4b Move description of the Rust flavor of stages to service.rs 2018-11-08 11:33:52 -07:00
Greg Fitzgerald f96563c3f2 Add documentation for pipelining 2018-11-08 11:33:52 -07:00
Rob Walker 30697f63f1
add support for slots in erasure (#1736) 2018-11-08 10:20:03 -08:00
Greg Fitzgerald 2a6046de8e
Cleanup TVU code to look like its block diagram (#1737)
* Reorg TVU code to look like TVU diagram

And move channel creation into LedgerWriteStage so that it can
be used in the same was as all the other stages.

* Delete commented out code
2018-11-07 19:25:36 -07:00
Greg Fitzgerald 43bd28cdfa Add loader_ prefix to LoaderTransaction methods 2018-11-07 15:06:38 -07:00
Rob Walker 6c10458b5b
leader slots in Blobs (#1732)
* add leader slot to Blobs
* remove get_X() methods in favor of X() methods for Blob
* add slot to get_scheduled_leader()
2018-11-07 13:18:14 -08:00
Stephen Akridge 7780d9bab8 Add ledger write and storage stage to TVU documentation 2018-11-07 12:07:12 -08:00
Greg Fitzgerald 038a46b5ef Integrate the markdown book into the codebase
This implies that the book should describe exactly what is implemented,
and will not lead the way and eventually bitrot as the RFCs do.
2018-11-07 10:58:47 -07:00
Greg Fitzgerald 8be7c13d2d
Stub out architecture book (#1674)
* Stub out architecture documentation

* Add book HTML generation and book tests to CI

* Add heading

* Better table of contents

* Reference existing documentation

Move ASCII art from code comments into rendered SVG

* Attempt to fix CI

* Add lamport docs

And truncate lines to 80 characters

* Fix links

And reference shorter, newer description of PoH.

* Replace ASCII art with SVG

* Streamline for Pillbox

* Update path before optional install

* Use $CARGO_HOME instead of $HOME

* Delete code

Attempt to describe all data structures without code.

* Boot RPU from docs, add JsonRpcService

Also, use Rust naming conventions in the block diagrams to
minimize the jump from docs to code.

* Latest code uses tick_height

* Rename bob/ folder to art/

A home for any ASCII art

* Import JSON RPC API

* More mdbook docs

* Add Ncp

* Cleanup links

* Move pipelining description into fullnode description

* Move high-level transaction docs into top-level doc

* Delete unused files
2018-11-06 18:00:58 -07:00
Tyera Eulberg d7ea66b6a1 RPC and Pubsub, bind to 0.0.0.0 2018-11-06 15:45:36 -07:00
Sathish 371c69d425
Add ledger write stage counters (#1713) 2018-11-06 14:44:54 -08:00
Rob Walker 9b43b00d5c
remove tick_count, leader_scheduler, from broadcast code (#1725) 2018-11-06 13:17:41 -08:00
Rob Walker 76694bfcf4
remove entry_writer.rs (#1720) 2018-11-06 12:42:31 -08:00
Greg Fitzgerald bfad138bb3 Pass any serializable to Transaction constructor 2018-11-06 11:23:59 -07:00
Greg Fitzgerald 54968b59bb Update last_id between client retries
Fixes #1694
2018-11-06 09:06:15 -07:00
Tyera Eulberg f683817b48 Remove RPU; replace with RPC 2018-11-05 20:30:47 -07:00
Rob Walker 22abc27be4
add tests for bank.purge() (#1711) 2018-11-05 16:43:27 -08:00
Greg Fitzgerald c9138f964b Change token type from i64 to u64
Fixes #1526
2018-11-05 15:25:26 -07:00
Rob Walker 1fbf1d2cf2
Add checkpoint, rollback to to bank (#1662)
add linked-list capability to accounts

change accounts from a linked list to a VecDeque

add checkpoint and rollback for lastids

add subscriber notifications for rollbacks

checkpoint transaction count, too
2018-11-05 09:47:41 -08:00
Rob Walker 1d7722043f genesis has 3 entries now 2018-11-02 22:02:13 -07:00
Rob Walker 95f9488a70
use default buffer size for index, use BLOB_DATA_SIZE for data buffer (#1693) 2018-11-02 21:52:57 -07:00
Greg Fitzgerald e7cbbd8d45 cargo fmt 2018-11-02 19:54:49 -07:00
Greg Fitzgerald c8c255ad73 Rename Budget to BudgetExpr 2018-11-02 19:54:49 -07:00
Michael Vines a264f8fa9b Fix |cargo test| 2018-11-02 19:04:59 -07:00
Michael Vines 40e945b0c8 Move token_program from src/ to programs/native/ 2018-11-02 18:13:02 -07:00
carllin 0636399b7a
Compute finality computation in new ComputeLeaderFinalityService (#1652)
* Move finality computation into a service run from the banking stage, ComputeLeaderFinalityService

* Change last ids nth to tick height, remove separate tick height from bank
2018-11-02 15:49:14 -07:00
carllin 298bd6479a
Add first leader to genesis (#1681)
* Add first leader to genesis entries, consume in genesis.sh

* Set bootstrap leader in the bank on startup, remove instantiation of bootstrap leader from bin/fullnode

* Remove need to initialize bootstrap leader in leader_scheduler, now can be read from genesis entries

* Add separate interface new_with_leader() in mint for creating genesis leader entries
2018-11-02 14:32:05 -07:00
Tyera Eulberg 879431ebcd Add timeout to TcpStream connect, and rename test 2018-11-01 14:13:19 -06:00
Tyera Eulberg af1283e92c Improve airdrop confirmation logic 2018-11-01 11:28:33 -06:00
Sathish dffa2eb04f
Do not parallelize deserialize operation (#1663)
Deserialize operations are faster when done serially with the
MT banking stage and helps with performance improvement with
reduced thread context switches.
2018-10-31 22:12:15 -07:00
carllin 00889c5139
Fix bad function arguments (#1682) 2018-10-31 19:55:58 -07:00
Stephen Akridge af8dc3fd83 Fix snap build
cuda and chacha features required for chacha_cuda
2018-10-31 17:59:31 -07:00
carllin ba884b4e36
Add thin client test for vote functionality, fix sizing errors in vote contract (#1643)
* Added tests to thin client to test VoteContract calls, fix VoteContract sizing errors

* Calculate upper bound on VoteProgram size at runtime, add test for serializing/deserializing a max sized VoteProgram state
2018-10-31 17:47:50 -07:00
Tyera Eulberg 6ddd494826 Improve rpc logging 2018-10-31 15:21:55 -06:00
Stephen Akridge aa2fd3f3bb Storage RFC grammar 2018-10-31 13:44:21 -07:00
Stephen Akridge cf00354f42 Add storage stage which does storage mining verification for validators 2018-10-31 13:44:21 -07:00
carllin 47f1fa3f2e
Remove purging of leader id from cluster info (#1642) 2018-10-31 12:30:48 -07:00
Michael Vines 74b4ecb7f3 Upgrade to influx_db_client@0.3.6 2018-10-30 19:44:09 -07:00
Michael Vines 8d16f69bb9 Improve account subscribe/unsubscribe logging 2018-10-30 12:03:35 -07:00
Michael Vines 3a73a09391 Avoid panicking when a native library doesn't exist 2018-10-30 12:03:35 -07:00
Michael Vines 009c71f7e2 Demote info logs 2018-10-30 12:03:35 -07:00
Michael Vines 073d39df44 Add solana_ prefix to loaders so their logs appear in the default RUST_LOG config 2018-10-30 12:03:35 -07:00
Michael Vines ae7222f0df Work around influxdb panic 2018-10-30 12:03:35 -07:00
Rob Walker 13bfdde228
remove ledger tail code, WINDOW_SIZE begone (#1617)
* remove WINDOW_SIZE, use window.window_size()
* move ledger tail, redundant with ledger-based repair
2018-10-30 10:05:18 -07:00
jackcmay be003970b7
Program_ids were overlapping (#1626)
Program_ids were overlapping
2018-10-26 19:44:53 -07:00
Stephen Akridge b28fbfa13e Use a smaller test value for window_size
Otherwise this test takes forever to run.
2018-10-26 11:38:55 -07:00
Michael Vines 07c656093c Remove tictactoe programs 2018-10-25 21:22:07 -07:00
Michael Vines c9e8346e6a
cargo fmt 2018-10-25 17:24:24 -07:00
Michael Vines 236113e417 cargo fmt 2018-10-25 17:13:41 -07:00
carllin f6c8e1a4bf
Vote contract (#1552)
* Add Vote Contract

* Move ownership of LeaderScheduler from Fullnode to the bank

* Modified ReplicateStage to consume leader information from bank

* Restart RPC Services in Leader To Validator Transition

* Make VoteContract Context Free

* Remove voting from ClusterInfo and Tpu

* Remove dependency on ActiveValidators in LeaderScheduler

* Switch VoteContract to have two steps 1) Register 2) Vote. Change thin client to create + register a voting account on fullnode startup

* Remove check in leader_to_validator transition for unique references to bank, b/c jsonrpc service and rpcpubsub hold references through jsonhttpserver
2018-10-25 16:58:40 -07:00
carllin 55833e20b1
Create Poh Service (#1604)
* Create new Poh Service, replace tick generation in BankingStage
2018-10-25 14:56:21 -07:00
Tyera Eulberg 02cfa76916 Plumb GetTransactionCount through solana-wallet 2018-10-25 14:58:51 -06:00
Tyera Eulberg 9314eea7e9 Add leader-readiness test to wallet-sanity 2018-10-25 14:58:51 -06:00
Michael Vines e47fcb196b s/solana_program_interface/solana[_-]sdk/g 2018-10-25 12:31:45 -07:00
Michael Vines af8a36b7fb Exclude chacha_cuda when chacha is disabled 2018-10-24 17:02:46 -07:00
Michael Vines 208e7d7943 Explicitly reject transactions larger than PACKET_SIZE 2018-10-24 15:34:27 -07:00
Stephen Akridge 61927e1941 Fix compile error for write_entries
Takes a reference now.
2018-10-24 11:31:30 -07:00
Tyera Eulberg 2f2531d921 Add retries to Wallet deploy 2018-10-24 11:13:32 -06:00
Tyera Eulberg d5f20980eb Incorporate preloaded bpf loader 2018-10-24 11:13:32 -06:00
Tyera Eulberg 21eae981f9 Add deploy method to solana-wallet 2018-10-24 11:13:32 -06:00
Stephen Akridge ead7f4287a Storage mining fixups...
* Use IV to make unique identies
* Use hex! macro for hex literal and not string converted to u8 slice
* fix sha sampling to control init/end of sha state
2018-10-24 09:58:41 -07:00
Michael Vines 3b33150cfb Bump drone read timeout to 10s
The previous timeout of 3s was not generous enough occasionally
2018-10-24 08:52:41 -07:00
Pankaj Garg 5c483c9928 remove unused variable 2018-10-23 16:52:56 -06:00
Pankaj Garg a68c99d782 Fix transaction count on testnet dashboard 2018-10-23 16:52:56 -06:00
jackcmay 5ca52d785c
Preload BPF loader (#1573)
Preload BPF loader
2018-10-23 14:44:41 -07:00
Pankaj Garg 8d9912b4e2
Move ledger write to its own stage (#1577)
* Move ledger write to its own stage

- Also, rename write_stage to leader_vote_stage, as write functionality
  is moved to a different stage

* Address review comments

* Fix leader rotation test failure

* address review comments
2018-10-23 14:42:48 -07:00
Michael Vines 7977b97227 Surface AccountInUse to JSON RPC users so they know to retry the transaction 2018-10-23 13:55:30 -07:00
Michael Vines 4f34822900 Improve logging on various error conditions 2018-10-23 13:40:59 -07:00
Pankaj Garg bbb38ac106
Increase window size (#1578)
Addresses the following problem
- Validators are not able to keep up with the leader
- The future blobs (outside of window) get dropped
- The validators won't process repair requests for these future blobs
2018-10-23 10:25:01 -07:00
Michael Vines 45cfa5b574 Add instruction to transfer account ownership 2018-10-20 21:54:25 -05:00
Pankaj Garg df9ccce5b2
Remove hostname() from calls to metrics as it's expensive operation (#1557) 2018-10-20 06:38:20 -07:00
jackcmay f8516b677a
Load program data in chunks (#1556)
Load program data in chunks
2018-10-19 18:28:38 -07:00
Greg Fitzgerald 26b99d3f85 Ensure witness and timestamp keys are signed
Before this patch, an attacker could point Budget instructions to
unsigned keys, and authorize a transaction from an unauthorized
party.
2018-10-19 10:06:59 -06:00