Commit Graph

534 Commits

Author SHA1 Message Date
carllin f0a40862d6
Support local cluster edge case testing (#7135)
* Refactor local cluster to support killing a partition

* Rework run_network_partition

* Introduce fixed leader schedule

* Plumb fixed schedule into test
2019-12-03 16:31:59 -08:00
Sunny Gleason 0a0f15baca
RPC subscriptions for new slot notifications (#7114)
* feat: slot notifications via pubsub rpc w/ tests
2019-11-26 03:42:54 -05:00
Tyera Eulberg 0f66e5e49b Add getConfirmedBlock test to rpc (#7120)
automerge
2019-11-25 11:08:03 -08:00
Michael Vines 0bd41f98ed Avoid jemalloc in windows build (#7089)
automerge
2019-11-21 18:39:29 -08:00
anatoly yakovenko d9e7a5fcbe
Use fork weight instead of individual bank weight for fork selection. (#7079)
* Fix weight calculation

* Fix tests

* fork weight

* wait until nodes are in the leader schedule

* enable sanity

* fewer long tests
2019-11-21 15:47:08 -08:00
Tyera Eulberg c965a110f2
Use unbounded channel (#7081) 2019-11-21 14:23:40 -07:00
sakridge e66b29943b
datapoint for best fork weight and slot in replay (#7066) 2019-11-20 17:26:52 -08:00
Tyera Eulberg 97ca6858b7
Write transaction status and fee into persistent store (#7030)
* Pass blocktree into execute_batch, if persist_transaction_status

* Add validator arg to enable persistent transaction status store

* Pass blocktree into banking_stage, if persist_transaction_status

* Add validator params to bash scripts

* Expose actual transaction statuses outside Bank; add tests

* Fix benches

* Offload transaction status writes to a separate thread

* Enable persistent transaction status along with rpc service

* nudge

* Review comments
2019-11-20 16:43:10 -07:00
anatoly yakovenko b150da837a Use epoch as the gossip purge timeout for staked nodes. (#7005)
automerge
2019-11-20 11:25:18 -08:00
Jack May d184d3a732
Merge native programs parts into one unit (#7047) 2019-11-20 10:12:43 -08:00
Pankaj Garg d32a072190
Use ticks_per_slot to calculate maximum grace ticks (#7024)
* Use ticks_per_slot to calculate maximum grace ticks

* fix test

* fix votable candidate ordering

* fixes to pick_best_fork() and a unit test

* fixes
2019-11-19 17:55:42 -08:00
Dan Albert b958bf9086
Fix confirmation metrics (#7035) 2019-11-19 09:51:50 -05:00
carllin 43144cfe8b
Make banks that fail threshhold check resettable (#7027) 2019-11-19 02:36:30 -08:00
carllin 11d2d2eccd
Fix progress map losing banks and recomputing stats (#7026)
* Fix progress map missing banks

* Fix confirmations

* Fix test

* Initialiize progress with frozen banks atartup
2019-11-19 02:36:00 -08:00
Sagar Dhawan 6bfe0fca1f
Add a version field to shreds (#7023)
* Add a version field to shreds

* Clippy

* Fix Chacha Golden

* Fix shredder bench compile

* Fix blocktree bench compile
2019-11-18 18:05:02 -08:00
carllin b66c03667c
Log for threshold failure (#7008) 2019-11-17 17:10:16 -08:00
carllin 70322d1ff8
Add error logging to dead slots (#7000) 2019-11-16 02:54:51 -08:00
anatoly yakovenko 59413b3124 Fix rules for fork selection (#6906)
automerge
2019-11-15 08:36:33 -08:00
Sagar Dhawan 79d7090867
Remove obsolete references to Blob (#6957)
* Remove the name "blob" from archivers

* Remove the name "blob" from broadcast

* Remove the name "blob" from Cluset Info

* Remove the name "blob" from Repair

* Remove the name "blob" from a bunch more places

* Remove the name "blob" from tests and book
2019-11-14 11:49:31 -08:00
carllin 43e2301e2c
Fix roots overrunning broadcast (#6884)
* Add trusted pathway for insert_shreds to avoid checks
2019-11-14 00:32:07 -08:00
Greg Fitzgerald a3a830e1ab
Delete Service trait (#6921) 2019-11-13 11:12:09 -07:00
Greg Fitzgerald 30a08f4282 Cleanup ledger macros (#6916)
automerge
2019-11-13 07:14:09 -08:00
Justin Starry 9807f47d4e
Rename genesis block to genesis config (#6816) 2019-11-08 23:56:57 -05:00
Justin Starry 02058ea699
Reject blocks with invalid last ticks in replay stage (#6833)
* Reject blocks with invalid last ticks in replay stage

* slot_full
2019-11-08 20:21:54 -05:00
Michael Vines b800642fa4
Add new fork log message for when the node is leader for consistency (#6808) 2019-11-08 12:30:25 -07:00
Pankaj Garg 0ace79939b
Add reference tick to data shreds (#6772)
* Add reference tick to data shreds

* fix tests
2019-11-06 13:27:58 -08:00
carllin 24102a7435
Allow voting on empty banks (#6719)
* Allow votes on empty banks

* Remove making first bank is_delta true, no longer necessary for idling

* Remove votable from ledger tool
2019-11-06 01:02:26 -08:00
Michael Vines 09e648f957
ledger-tool/: Include full validator voting history in fork-graph (#6756) 2019-11-05 19:40:00 -07:00
anatoly yakovenko f54cfcdb8f
Store and persists full stack of tower votes in gossip (#6695)
* vote array

wip

wip

wip

update

gossip index should match tower index

tests build

clippy

test index after expired vote

test

bank specific last vote sync time

* verify

* we are likely to see many more warnings about old votes now
2019-11-04 16:19:54 -08:00
sakridge 57983980a7 Lower verify-batch-size to debug (#6722)
automerge
2019-11-04 16:00:59 -08:00
Tyera Eulberg 33f4aaf3fd
Rename confidence to commitment (#6714) 2019-11-04 16:44:27 -07:00
Michael Vines 50a17fc00b Use Slot and Epoch type aliases instead of raw u64 (#6693)
automerge
2019-11-02 00:38:30 -07:00
Justin Starry e8e5ddc55d
Verify number of hashes for each block of entries (#6262)
* Verify number of hashes for each block of entries

* Fix blocktree processor tick check

* Rebase once more
2019-10-31 16:38:50 -04:00
sakridge 38491c8c4b
Reduce verify-batch-size log (#6623) 2019-10-30 13:41:11 -07:00
Rob Walker b4119c454a
credit_only credits forwarding (#6509)
* credit_only_credits_forwarding

* whack transfer_now()

* fixup

* bench should retry the airdrop TX

* fixup

* try to make bench-exchange a bit more robust, informative
2019-10-23 22:01:22 -07:00
sakridge 03d29a8311
Async poh verify (#6353)
* Async poh verify

* Up ticks_per_s to 160

GPU poh verify needs shorter poh sequences or it takes forever to
verify. Keep slot time the same at 400ms.

* Fix stats

* Don't halt on ticks

* Increase retries for local_cluster tests and make repairman test serial
2019-10-23 12:11:04 -07:00
carllin b38bf90de7
Deshred blocks in parallel (#6461)
* Deshred in parallel

* Add tests for corrupt slots and parallel deshred

* Rename load_blocktree_entries to load_blocktree_entries_with_shred_count
2019-10-21 16:15:10 -07:00
Pankaj Garg 84e911361a
Use constants instead of lazy_static for shred header sizes (#6472) 2019-10-21 12:46:16 -07:00
Greg Fitzgerald 2636418659
Move blocktree_processor to solana_ledger (#6460)
* Drop core::result dependency in bank_forks

* Move blocktree_processor into solana_ledger
2019-10-20 09:54:38 -06:00
Rob Walker e2c316d2d0
system_instruction_processor updates (#6448)
* zero lamport account creation

* whack create_user_account, take 2

* target->to

* ..

* ..

* update chacha golden

* update chacha golden

* ..

* ..
2019-10-19 18:23:27 -07:00
Pankaj Garg badeb4d31a
Rework shred headers to fix position of signature (#6451)
* Rework shred headers to fix position of signature

* fix clippy
2019-10-18 22:55:59 -07:00
Greg Fitzgerald 5468be2ef9 Add solana-ledger crate (#6415)
automerge
2019-10-18 09:28:51 -07:00
carllin 27c0d30a07
Fix logging (#6417) 2019-10-18 02:06:41 -07:00
Greg Fitzgerald d3068c3918
Remove circular dependencies in core (#6408)
* Remove core::result dependency from blocktree

* Remove core::result dependency from shred

* Move Packet from core::packet to sdk::packet

This way we don't need to split perf_libs yet.

* Disable packet when compiling BPF programs
2019-10-17 11:37:30 -06:00
Pankaj Garg 83c1831a01
Fix replay stage test (#6406) 2019-10-16 15:41:43 -07:00
carllin ccb4e32ee0
ReplayStage metrics (#6358)
* ReplayStage metrics

* Add more metrics

* Refactor get_slot_entries_with_shred_count() to detect wasted work

* Update dashboard

* Update broadcast slots to micros

* Add broadcast dashboard
2019-10-16 14:32:18 -07:00
Justin Starry 7e6e7e8406
Remove special handling of first ledger tick (#6263)
* Remove special handling of first ledger tick

* Fix subtraction overflow

* @garious feedback

* Back to height

* More tick_height name changes

* Fix off-by-one

* Fix leader tick error

* Fix merge conflict

* Fix recently added test
2019-10-16 15:53:11 -04:00
sakridge dfca2b510b
Lower shred/receiver stats (#6365)
too many messages
2019-10-15 11:43:52 -07:00
Rob Walker 7cf90766a3
add epoch_schedule sysvar (#6256)
* add epoch_schedule sysvar

* book sheesh!
2019-10-08 22:34:26 -07:00
Justin Starry 5e31565574
Expand blocktree processor options (#6248)
* Refactor blocktree processor args and support full leader cache

* Add entry callback option

* Rename num_threads to override_num_threads

* Add test for entry callback

* Refactor cached leader schedule changes

* Add tests for blocktree process options

* Refactor test

* @mvines feedback
2019-10-08 17:58:49 -04:00
sakridge 5ef012b2c1
Tweak debug to remove unreadable datapoints (#6060) 2019-10-04 16:25:22 -07:00
Michael Vines f9f5bc2eb5
More clippy 2019-10-02 21:21:07 -07:00
Sagar Dhawan 84f74807d4 Skip considering banks older than the latest vote slot (#6037)
automerge
2019-09-23 19:40:03 -07:00
Sagar Dhawan 62c22c6cb1
Fix really old banks triggering log spam (#6025) 2019-09-23 13:59:16 -07:00
carllin 8240d1fe0a
Confidence implementation (#5993)
* Change confidence parameters

* Add status_cache_ancestors to get all relevant ancestors of a bank including roots from status cache

* Fix and add tests

* Clippy
2019-09-20 19:38:56 -07:00
Pankaj Garg ca9d4e34df
Broadcast stage tuning (#5989) 2019-09-19 16:29:52 -07:00
Pankaj Garg 783e8672e7
Removed Shred enum (#5963)
* Remove shred enum and it's references

* rename ShredInfo to Shred

* clippy
2019-09-18 16:24:30 -07:00
carllin 2f50d0e145
Refactor confidence from replay stage (#5938) 2019-09-17 19:43:40 -07:00
Pankaj Garg ff608992ee
Replace Shred usage with ShredInfo (#5939)
* Replace Shred usage with ShredInfo

* Fix tests

* fix clippy
2019-09-17 18:22:46 -07:00
Michael Vines 84c8a5bbec
Add replay-stage-mark_dead_slot datapoint (#5907) 2019-09-14 08:50:53 -07:00
Pankaj Garg 3d3b03a123
Verify signature of recovered shred before adding them to blocktree (#5811)
* Verify signature of recovered shred before adding them to blocktree

* fix failing tests, and review comments
2019-09-05 18:20:30 -07:00
carllin bd74e63702
Offload remaining confidence cache computation to separate thread (#5792)
* Move remaining confidence cache computation to separate thread

* Move confidence cache out of bank forks
2019-09-04 23:10:25 -07:00
Pankaj Garg 02ee2a601c
Further cleanup of blocktree after Blob deprecation (#5780) 2019-09-04 12:47:09 -07:00
carllin dfa6238342
Remove unnecessary construction of descendants (#5742) 2019-09-04 01:49:42 -07:00
Pankaj Garg 3b0d48e3b8
Remove blocktree blob references (#5691)
* Remove blocktree blob references

* fixes and cleanup

* replace uninitialized() call with MaybeUninit

* fix bench
2019-09-03 21:32:51 -07:00
anatoly yakovenko 475f6fe666
votes only need slots and the last bank hash (#5499)
churn

cleanup

reverse test slot hashes

test check_slots_are_valid

updates

only send the minimum bank vote difference

fixup! only send the minimum bank vote difference

some banks may not have a voting account setup

fixup! votes only need slots and the last bank hash

fixup! fixup! votes only need slots and the last bank hash

fmt

fixed compare

fixed vote

fixup! fixed vote

poke ci

filter the local votes via the last bank vote
2019-09-02 12:01:09 -07:00
Parth 7dfb735db9
randomize tx ordering (#4978)
Summary of Changes:
This change adds functionality to randomize tx execution for every entry. It does this by implementing OrderedIterator that iterates tx slice as per the order specified. The order is generated randomly for every entry.
2019-08-28 21:08:32 +05:30
carllin 8b9c3a2561
Blocktree last_root to enforce a slot floor (#5593)
* Add last_root to blocktree

* Don't repair earlier than last_root

* Add integration test to make sure blocktree floor is enforced
2019-08-27 15:09:41 -07:00
Pankaj Garg 4ac1213c9c
Integrate coding shreds and recovery (#5625)
* Integrate coding shreds and recovery

* More tests for shreds and some fixes

* address review comments

* fixes to code shred generation

* unignore tests

* fixes to recovery
2019-08-26 18:27:45 -07:00
Michael Vines faafee6b42
to to/the the (#5590) 2019-08-21 17:46:59 -07:00
Pankaj Garg 4798e7fa73
Integrate data shreds (#5541)
* Insert data shreds in blocktree and database

* Integrate data shreds with rest of the code base

* address review comments, and some clippy fixes

* Fixes to some tests

* more test fixes

* ignore some local cluster tests

* ignore replicator local cluster tests
2019-08-20 17:16:06 -07:00
Michael Vines ccee6241a6
Revert "publish fixes"
This reverts commit 4d13d3871d.
2019-08-16 17:28:07 -07:00
Michael Vines 4d13d3871d
publish fixes 2019-08-16 17:03:57 -07:00
Rob Walker 94f1132fb6
fix single node testnet, remove bootstrap vote (#5534) 2019-08-15 18:58:46 -07:00
Rob Walker d22848f9b1
use live stakes for consensus (#5426)
* use live stakes for consensus

* lint

* re-enable leader_failure_4

* fixups

* re-ignore leader_failure_4
2019-08-14 13:30:21 -07:00
carllin cd14a940d8
Allow process_blocktree() to start processing from any root (#5484)
* Remove unnecessary entry_height from BankInfo

* Refactor process_blocktree to support process_blocktree_from_root

* Refactor to process blocktree after loading from snapshot

* On restart make sure bank_forks contains all the banks between the root and the tip of each fork, not just the head of each fork

* Account for 1 tick_per_slot in bank 0 so that blockhash of bank0 matches the tick
2019-08-13 17:20:14 -07:00
Rob Walker 07d11be6ab
add global stake warmup (#5483)
* add global stake warmup

* integrate stake history into runtime

* fixup core tests

* fixup

* remove existing cooldown tests for now
2019-08-12 20:59:57 -07:00
Justin Starry 0fde19239b
Rate limit counter metrics points to one per second (#5496)
* Rate limit counter metrics points to one per second

* Remove old env var

* Test that metrics counter is incrementing

* Fix typo
2019-08-12 18:15:34 -04:00
Justin Starry 4f86c0b74a
Rate limit transaction counters (#5447)
* Rate limit transaction counters

* @sakridge feedback

* Set default high metrics rate for multinode demo

* Fix tests

* Swap defaults and fix env var tests

* Only set metrics rate if not already set
2019-08-08 17:05:06 -04:00
Tyera Eulberg efd09ecd37 Revert fork metrics (#5427)
* Revert "Remove duplicate row (#5419)"

This reverts commit a81dd80d60.

* Revert "Log fork stake-percentage in metrics, and display (#5406)"

This reverts commit 92e419f1c7.
2019-08-05 15:53:36 -07:00
Tyera Eulberg 92e419f1c7
Log fork stake-percentage in metrics, and display (#5406)
* Log fork stake percentage data

* Add fork stake percentage to dashboard

* Call out parent slot
2019-08-02 19:16:23 -06:00
Michael Vines eb3991b9ba
Replay stage log message nits (#5408) 2019-08-02 10:08:42 -07:00
carllin 6cb2040a1b
Snapshot Packaging Service (#5262)
* Snapshot serialization and packaging
2019-07-31 17:58:10 -07:00
Justin Starry 84368697af
Fix metrics when leader does not report metrics (#5291) 2019-07-30 16:18:33 -04:00
Pankaj Garg aef7bae60d
Let grace ticks to roll over into multiple leader slots (#5268)
* Let grace ticks to roll over into multiple leader slots

* address review comments
2019-07-26 11:33:51 -07:00
Tyera Eulberg dab7de7496
Add confidence cache to BankForks (#5066)
* Add confidence cache to BankForks

* Include stake-weighted lockouts in cache

* Add cache test

* Move confidence cache updates to handle_votable_bank

* Prune confidence cache on prune_non_root()

* Spin thread to process aggregate_stake_lockouts

* Add long-running thread for stake_weighted_lockouts computation
2019-07-26 11:27:57 -06:00
Justin Starry 547a7a345f
Add logs to indicate when the leader changes (#5253) 2019-07-23 22:19:20 -04:00
Michael Vines 0d8f5379a0
Add time units 2019-07-22 08:22:21 -07:00
Sagar Dhawan a07b17b9b5
Drop older slots in the ledger (#5188)
* Add facility to delete blocktree columns in range

* Add ledger cleanup service

* Add local_cluster test
2019-07-20 13:13:55 -07:00
Rob Walker afa05acb32
more replay_stage grooming (#5163) 2019-07-18 14:54:27 -07:00
Sagar Dhawan a3a91ba222 Fix misleading variable name (#5176)
automerge
2019-07-18 14:07:32 -07:00
Sagar Dhawan 751b54b60b
Skip sleeping in replay stage if a bank was recently processed (#5161)
* Skip sleeping in replay stage if a bank was recently processed

* Remove return
2019-07-18 12:04:53 -07:00
Rob Walker 0ffd91df27
groom poh_recorder (#5127)
* groom poh_recorder

* fixup

* nits

* slot() from the outside means "the slot the recorder is working on"

* remove redundant check

* review comments, put next_tick back in the "is reset" check

* remove redundant check
2019-07-17 14:10:15 -07:00
sakridge 40faaef9da
Revert "Logging (#5017)" (#5096)
This reverts commit b50a3bae72.
2019-07-14 18:48:15 -07:00
Pankaj Garg 4c90898f0b
Dynamic erasure set configuration (#5018)
* Use local erasure session to create/broadcast coding blobs

* Individual session for each recovery (as the config might be different)

* address review comments

* new constructors for session and coding generator

* unit test for dynamic erasure config
2019-07-11 13:58:33 -07:00
carllin b50a3bae72
Logging (#5017)
* Add logging to replay_stage

* locktower logging
2019-07-10 15:52:31 -07:00
Rob Walker f537482c86
remove set_leader from cluster_info (#4998) 2019-07-09 22:06:47 -07:00
Rob Walker f777a1a74c
groom replay_stage and poh_recorder (#4961)
* groom replay_stage and poh_recorder

* fixup

* fixup

* don't freeze() parent, need to review bank_forks and maybe vote...
2019-07-09 15:36:30 -07:00
Rob Walker eb4edd75e6
make bank commit_credits non public (#4944)
* make bank commit_credits non pub

* track down create() failure

* move bank_client to process_transaction(), which commits credits
2019-07-08 15:37:54 -07:00
carllin 2aac094f63
Ensure blobs are deserializable without unwrapping (#4948)
* Return result from deserializing blobs in blocktree instead of assuming deserialization will succeed

* Mark bad deserialization as dead fork

* Add test for corrupted blobs in blocktree and replay_stage
2019-07-07 14:37:12 -07:00
Tyera Eulberg 66552d7047
Credit-Only Accounts: Cache account balance for thread-safe load/store (#4691)
* Implement CreditOnlyLocks

* Update credit-only atomic on account load

* Update credit-only atomic after bank.freeze_lock; store credits if all credit-only lock references are dropped

* Commit credit-only credits on bank freeze

* Update core to CreditAccountLocks

* Impl credit-only in System Transfer

* Rework CreditAccountLocks, test, and fix bugs

* Review comments: Pass CreditAccountLocks by reference; Tighten up insert block

* Only store credits on completed slot

* Check balance in bench_exchange funding to ensure commit_credits has completed

* Add is_debitable info to KeyedAccount meta to pass into programs

* Reinstate CreditOnlyLocks check on lock_account

* Rework CreditAccountLocks to remove strong_count usage

* Add multi-threaded credit-only locks test

* Improve RwLocks usage

* Review comments: panic if bad things happen; tighter code

* Assert lock_accounts race does not happen

* Revert panic if bad things happen; not a bad thing
2019-06-27 17:25:10 -04:00
carllin d6737b8cc9
Set epoch schedule in set_root in leader schedule cache (#4821) 2019-06-26 00:19:48 -07:00
anatoly yakovenko 29611fb61d
tower consensus naming (#4598)
s/locktower/tower/g
2019-06-24 13:41:23 -07:00
carllin aacb38864c
Mark dead forks in replay stage (#4715)
* Add DeadSlots column family

* Filter dead forks from get_slots_since

* Mark erroring slots as dead in replay stage, add test

* Mark dead forks in progress instead of removing them

* Fix logging process_entries failures in replay_stage

* Unignore test_fail_entry_verification_leader
2019-06-20 15:50:41 -07:00
Sagar Dhawan 8019bff391 Fixes for storage program and rework storage stage (#4654)
automerge
2019-06-11 18:27:47 -07:00
Rob Walker 3217a1d70c
use highest staked node as bootstrap leader, remove bootstrap_leader from genesis_block (#4635)
* use highest staked node as bootstrap leader, remove bootstrap_leader from genesis_block

* clippy

* fixup

* fixup
2019-06-11 11:44:58 -07:00
Rob Walker 028e111fbc
remove payer from vote instructions (#4475) 2019-05-31 11:45:17 -07:00
Sathish 182096dc1a
Create bank snapshots (#4244)
* Revert "Revert "Create bank snapshots (#3671)" (#4243)"

This reverts commit 81fa69d347.

* keep saved and unsaved copies of status cache

* fix format check

* bench for status cache serialize

* misc cleanup

* remove appendvec storage on purge

* fix accounts restore

* cleanup

* Pass snapshot path as args

* Fix clippy
2019-05-30 21:31:35 -07:00
carllin 534244b322
Fix set_roots to use cached bank parents instead of searching blocktree (#4466) 2019-05-29 09:43:22 -07:00
carllin d772a27936 Plumb ClusterInfoRepairListener (#4428)
automerge
2019-05-24 19:20:09 -07:00
Michael Vines cfe5afd34c
_id => _pubkey variable renaming (#4419)
* wallet: rename *_account_id to *_account_pubkey

* s/from_id/from_pubkey/g

* s/node_id/node_pubkey/g

* s/stake_id/stake_pubkey/g

* s/voter_id/voter_pubkey/g

* s/vote_id/vote_pubkey/g

* s/delegate_id/delegate_pubkey/g

* s/account_id/account_pubkey/g

* s/to_id/to_pubkey/g

* s/my_id/my_pubkey/g

* cargo fmt

* s/staker_id/staker_pubkey/g

* s/mining_pool_id/mining_pool_pubkey/g

* s/leader_id/leader_pubkey/g

* cargo fmt

* s/funding_id/funding_pubkey/g
2019-05-23 23:20:04 -07:00
Michael Vines 2ed77b040a
create_genesis_block() now returns a struct (#4403) 2019-05-22 20:39:00 -07:00
Rob Walker 578c2ad3ea
add bank hash to votes (#4381) 2019-05-21 21:45:38 -07:00
carllin f1e5edee14
Modify Roots Column To Support Multiple Roots (#4321)
* Fix 1) Roots column family to handle storing multiple slots, 2) Store all slots on the rooted path in the roots column family
2019-05-20 19:04:18 -07:00
Sagar Dhawan af82b0dce9 Fix sending root slots instead of current slot (#4354)
automerge
2019-05-20 15:01:55 -07:00
Rob Walker 86e03a6d1b
support issuing vote instructions from system account (#4338)
* issue vote instructions from system account

* fixup

* bring back KeypairUtil
2019-05-20 13:32:32 -07:00
Sagar Dhawan 06eb2364f2
Handle missed slots in storage stage (#4337)
* Handle missed slots in storage stage

* Fix test compile

* Make test use the new missed slot support
2019-05-18 15:24:50 -07:00
Pankaj Garg 9476fe5ce3
Use log levels for datapoint metrics (#4335)
* Use log levels for datapoint metrics

* address review comments

* fix cyclomatic complexity
2019-05-17 17:34:05 -07:00
Pankaj Garg b2ce5dc9f5
Adjust log level for counter metrics (#4323) 2019-05-17 07:00:06 -07:00
Sagar Dhawan fd3f2cb910
Add Storage accounts for all nodes (#4298)
* Setup storage keypairs for all nodes

* Clean up naming

* clippy

* Update arg value_names
2019-05-15 15:19:29 -07:00
Jack May f567877d1d
Cleanup metrics (#4230) 2019-05-10 08:33:58 -07:00
Rob Walker 81fa69d347
Revert "Create bank snapshots (#3671)" (#4243)
This reverts commit abf2b300da.
2019-05-09 19:27:27 -07:00
Sathish abf2b300da Create bank snapshots (#3671)
* Be able to create bank snapshots

* fix clippy

* load snapshot on start

* regenerate account index from the storage

* Remove rc feature dependency

* cleanup

* save snapshot for slot 0
2019-05-09 19:27:06 -07:00
Rob Walker 8e400fc4bd
rework genesis (passive staking groundwork) (#4187)
* rework genesis

* fixup
2019-05-07 11:16:22 -07:00
Tyera Eulberg 71f9b44687
Add Confirmations parameter to RPC Subscriptions (#4154)
* Add optional depth parameter to pubsub, and store in subscriptions

* Pass bank_forks into rpc_subscription; add method to check depth before notify and impl for account subscriptions

* Impl check-depth for signature subscriptions

* Impl check-depth for program subscriptions

* Plumb fork id through accounts

* Use fork id and root to prevent repeated account notifications; also s/Depth/Confirmations

* Write tests in terms of bank_forks

* Fixup accounts tests

* Add pubsub-confirmations tests

* Update pubsub documentation
2019-05-06 08:31:50 -06:00
Sagar Dhawan a7b695c27a
Change replicators to slot-based (#4118) 2019-05-03 16:27:53 -07:00
carllin 1a259d4a3f Prevent Requests/Caching of leader schedules for epochs beyond confirmed roots (#4033)
automerge
2019-04-30 13:23:21 -07:00
carllin 512bfc93cb
Add a cache for leader schedules (#3841)
* Add a cache for leader schedules
2019-04-19 02:39:44 -07:00
Rob Walker 64c6f05da2
persist set_root() and use it in blocktree_processor to limit squashes (#3782)
* rename locktower's slot to epoch

* persist set_root() and use it in blocktree_processor to limit squashes
2019-04-15 13:12:28 -07:00
Sagar Dhawan 1db80d79fc Update get recent blockhashes to return confirmed blockhashes only 2019-04-12 14:58:22 -07:00
Sagar Dhawan 877ec08280
Send recent votes in Vote Transactions (#3734) 2019-04-11 14:48:36 -07:00
Rob Walker f1e7237c09
vote_api cleanup (#3710)
* vote_api cleanup

* fixups

* fixup

* remove unused code

* revert removal of serialize and deserialize

* ...

* increase coverage, bootstrap staking

* Sagar's STAKE to my VOTE
2019-04-10 17:52:47 -07:00
carllin d4eebcc2aa
Check for frozen in confirm_forks (#3678) 2019-04-09 11:45:38 -07:00
carllin 20aa4434e2
Fix repair (#3581)
Add DetachedHeads repair protocol

Add DetachedHeads repair test

Repair starting from root
2019-04-06 19:41:22 -07:00
Rob Walker a5b5248a09
move vote_accounts up (#3647) 2019-04-05 14:23:00 -07:00
Greg Fitzgerald 35298e01a8 Remove Instruction wrapper structs and name functions after enum fields 2019-04-03 13:34:27 -07:00
Greg Fitzgerald fcef54d062 Add a constructor to generate random pubkeys 2019-03-31 16:23:18 -06:00
carllin f886b3b12b
Fix resetting PohRecorder to wrong bank (#3553)
* Check whether future slot already has transmission
2019-03-29 20:00:36 -07:00
Greg Fitzgerald 5646daa820 Delete lots of fee parameters
So many zeros!
2019-03-29 19:21:51 -06:00
carllin dee5ede16d
Get rid of unnecessary frozen banks (#3572) 2019-03-29 16:06:48 -07:00
Anatoly Yakovenko e45f7afd85 use the right id for delegate id 2019-03-27 15:04:09 -07:00
Sagar Dhawan 36ea088387
Fix Storage Stage not receiving entries when node is leader (#3528) 2019-03-27 13:10:33 -07:00
Anatoly Yakovenko 0346b9cb5c hang out on progress until fork is confirmed 2019-03-27 08:41:41 -07:00
Anatoly Yakovenko 6956bf635e validator confirmaiton 2019-03-27 08:41:41 -07:00
Anatoly Yakovenko e27d6d0988 validator confirmation 2019-03-27 08:41:41 -07:00
Greg Fitzgerald 4bc3f70150 Boot VoteTransaction 2019-03-25 17:11:57 -07:00
anatoly yakovenko 51004881f8
filter out banks that have an older epoch (#3472) 2019-03-25 11:09:39 -07:00
Pankaj Garg 46ecac3310
Update leader slot in poh recorder if we skipped it (#3452)
* reset poh recorder with the original start slot
2019-03-23 13:07:09 -07:00
Michael Vines f09b8d3921 Demote log level 2019-03-22 17:00:09 -07:00
Sathish 06b0c98c75
Remove accounts when the fork is removed (#3384)
* Fix test

* Cleanup accounts when the fork is removed

* Update test to check for deleted accounts
2019-03-21 17:36:10 -07:00
Stephen Akridge 412ebfcaf2 ReplayStage::new is too long
break into more functions
2019-03-21 14:08:11 -07:00
Greg Fitzgerald 07d55d0092 Downgrade 'No next leader found' to warning 2019-03-21 11:18:49 -06:00
anatoly yakovenko 148e08a8a5
Enable cluster tests (#3372)
* Cluster tests

* stable!

* fixup! stable!

* fixup! fixup! stable!

* fixup! fixup! fixup! stable!

* fixup! fixup! fixup! fixup! stable!

* fixed space

* add getNumBlocksSinceSignatureConfirmation entry for the json rpc docs

* Check in upcoming epochs for potential leadership slots in next_leader_slot()
2019-03-21 07:43:21 -07:00
carllin fe9e771b9b
Clear progress map on squash (#3377) 2019-03-19 17:30:36 -07:00
Anatoly Yakovenko 7246d72f03 fix is_locked_out logic 2019-03-19 16:21:46 -07:00
Carl 5e21268ca0 PR comments 2019-03-18 20:46:11 -07:00
Carl b38e3bef01 Modify bank_forks to support squashing/filtering new root and also don't remove parents from bank_forks when inserting, otherwise we lose potential fork points when querying blocktree for child slots 2019-03-18 20:46:11 -07:00
Stephen Akridge ef111dcbe1 Decendent is not a word 2019-03-18 15:58:27 -07:00
Pankaj Garg efc39ffdde
Report how many grace ticks were afforded to previous leader (#3350) 2019-03-18 13:24:07 -07:00
anatoly yakovenko 61a4b998fa
Implement locktower voting (#3251)
* locktower components and tests

* integrate locktower into replay stage

* track locktower duration

* make sure threshold is checked after simulating the vote

* check vote lockouts using the VoteState program

* duplicate vote test

* epoch stakes

* disable impossible to verify tests
2019-03-18 12:12:33 -07:00
Sagar Dhawan 00f005af25 Fix leader rotation counter 2019-03-15 17:01:18 -07:00
Pankaj Garg 971da7325d Reduce log level for periodic debug messages 2019-03-15 15:41:26 -07:00
Pankaj Garg 66fb1bbb2e
Give last leader some grace ticks to catch up (#3299)
* Wait for last leader for some ticks

* New tests and fixed existing tests
2019-03-15 13:22:16 -07:00
Rob Walker 9f02a8d3d0
remove ticks_per_slot from blocktree (#3297) 2019-03-14 15:18:37 -07:00
carllin af03df38b9
Don't vote for empty leader transmissions (#3248)
* Don't vote for empty leader transmissions

* Add is_delta flag to bank to detect empty leader transmissions

* Plumb new is_votable flag through replay stage

* Fix PohRecorder tests

* Change is_delta to AtomicBool to avoid making Bank references mutable

* Reset start slot in poh_recorder when working bank is cleared, so that connsecutive TPU's will start from the correct place

* Use proper max tick height calculation

* Test for not voting on empty transmission

* tests for is_votable
2019-03-13 14:06:12 -07:00
carllin cb3eeace56
Replay Stage start_leader() can use wrong parent fork() (#3238)
*  Make sure start_leader starts on the last voted block, not necessarily the biggest indexed bank in frozen_slots()

* Fix tvu test
2019-03-12 17:42:53 -07:00
Pankaj Garg b418525464 Update current leader information in metrics and dashboard 2019-03-11 17:43:59 -07:00
anatoly yakovenko 0c592c52f6
Wake up replay stage when the poh bank is cleared. (#3211)
* wake up replay stage when the poh bank is cleared

* bump ticks per second

* Increase ticks per slot to match faster tick rate

* Remove check that working bank must be the bank for the greatest slot

* Make start_leader() skip starting TPU for slots we've already been leader for
2019-03-11 13:58:23 -07:00
Rob Walker 195a880576
pass Pubkeys as refs, copy only where values needed (#3213)
* pass Pubkeys as refs, copy only where values needed

* Pubkey is pervasive

* fixup
2019-03-09 19:28:43 -08:00
Rob Walker 0acdbc0d03 plumb staking_account and voting_keypair from multinode-demo to Vote (#3199)
* plumb staking_account and voting_keypair from bash to Vote
2019-03-08 19:29:08 -07:00
Sagar Dhawan 02eb234399
Fix TVU and PoH Recorder going out of sync (#3164)
* Fix broadcast_stage error

* Account for very fast ticks in tick verification
2019-03-07 15:49:07 -08:00
Michael Vines a3cab470d3 Rename ClusterInfo::new_with_keypair() to ClusterInfo::new() 2019-03-07 13:05:42 -08:00
Michael Vines bb93504965 Rename ClusterInfo::new() to ClusterInfo::new_with_invalid_keypair() 2019-03-07 13:05:42 -08:00
anatoly yakovenko 1c0cfb17a3 Start leader based on Poh tick height. (#3084)
* Start leader based on poh and test

* Equalize validator and leader stakes in LocalCluster

* Clear WorkingBank on poh_recorder reset
2019-03-05 17:56:51 -08:00
Michael Vines 6ab6e6cb9b Clean up exit flag handing across TVU 2019-03-04 21:26:50 -08:00
Rob Walker 794e961328
use Bank's notion of leader_id where possible (#3119) 2019-03-04 18:40:47 -08:00
Rob Walker 6d82123125 rename bank_id to bank_slot 2019-03-04 17:10:27 -08:00
anatoly yakovenko 1654199b23
Use PohRecorder to synchronize instead of rotate. (#3080) 2019-03-03 16:44:06 -08:00
Greg Fitzgerald 1edf6c361e
Move Vote program out of the SDK 2019-03-02 16:44:36 -08:00
Michael Vines a94880574b block_hash => blockhash 2019-03-02 12:13:30 -07:00
Michael Vines 0f1582c196 cargo fmt 2019-03-02 12:13:30 -07:00
Michael Vines 258cf21416 Purge remaining last_id (now called block_hash) 2019-03-02 12:13:30 -07:00
Michael Vines 2bfad87a5f Rename Bank.last_id() to Bank.last_block_hash() 2019-03-02 12:13:30 -07:00
Michael Vines 5f5d779ee1 Move src/ into core/src. Top-level crate is now called solana-workspace 2019-03-02 09:52:18 -08:00