Commit Graph

110 Commits

Author SHA1 Message Date
carllin 55fc963595
Move slot cleanup to AccountsBackgroundService (#13911)
* Move bank drop to AccountsBackgroundService

* Send to ABS on drop instead, protects against other places banks are dropped

* Fix Abi

* test

Co-authored-by: Carl Lin <carl@solana.com>
2020-12-13 01:22:34 +00:00
Tyera Eulberg 89621adca7
Rpc -> proper optimistic confirmation (#12514)
* Add service to track the most recent optimistically confirmed bank

* Plumb service into ClusterInfoVoteListener and ReplayStage

* Clean up test

* Use OptimisticallyConfirmedBank in RPC

* Remove superfluous notifications from RpcSubscriptions

* Use crossbeam to avoid mpsc recv_timeout panic

* Review comments

* Remove superfluous last_checked_slots, but pass in OptimisticallyConfirmedBank for complete correctness
2020-09-28 20:43:05 -06:00
carllin 06f84c65f1
Fix rooted accounts cleanup, simplify locking (#12194)
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-28 16:04:46 -07:00
Ryo Onodera cb8661bd49
Persistent tower (#10718)
* Save/restore Tower

* Avoid unwrap()

* Rebase cleanups

* Forcibly pass test

* Correct reconcilation of votes after validator resume

* d b g

* Add more tests

* fsync and fix test

* Add test

* Fix fmt

* Debug

* Fix tests...

* save

* Clarify error message and code cleaning around it

* Move most of code out of tower save hot codepath

* Proper comment for the lack of fsync on tower

* Clean up

* Clean up

* Simpler type alias

* Manage tower-restored ancestor slots without banks

* Add comment

* Extract long code blocks...

* Add comment

* Simplify returned tuple...

* Tweak too aggresive log

* Fix typo...

* Add test

* Update comment

* Improve test to require non-empty stray restored slots

* Measure tower save and dump all tower contents

* Log adjust and add threshold related assertions

* cleanup adjust

* Properly lower stray restored slots priority...

* Rust fmt

* Fix test....

* Clarify comments a bit and add TowerError::TooNew

* Further clean-up arround TowerError

* Truly create ancestors by excluding last vote slot

* Add comment for stray_restored_slots

* Add comment for stray_restored_slots

* Use BTreeSet

* Consider root_slot into post-replay adjustment

* Tweak logging

* Add test for stray_restored_ancestors

* Reorder some code

* Better names for unit tests

* Add frozen_abi to SavedTower

* Fold long lines

* Tweak stray ancestors and too old slot history

* Re-adjust error conditon of too old slot history

* Test normal ancestors is checked before stray ones

* Fix conflict, update tests, adjust behavior a bit

* Fix test

* Address review comments

* Last touch!

* Immediately after creating cleaning pr

* Revert stray slots

* Revert comment...

* Report error as metrics

* Revert not to panic! and ignore unfixable test...

* Normalize lockouts.root_slot more strictly

* Add comments for panic! and more assertions

* Proper initialize root without vote account

* Clarify code and comments based on review feedback

* Fix rebase

* Further simplify based on assured tower root

* Reorder code for more readability

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-19 14:03:54 +09:00
Tyera Eulberg 05db41fe9c
Cache block time in Blockstore (#11955)
* Add blockstore column to cache block times

* Add method to cache block time

* Add service to cache block time

* Update rpc getBlockTime to use new method, and refactor blockstore slightly

* Return block_time with confirmed block, if available

* Add measure and warning to cache-block-time
2020-09-09 09:33:14 -06:00
carllin 1c1a3f979d
Detect and notify when deserializable shreds are available (#11816)
* Add logic to check for complete data ranges

* Add RPC signature notification

Co-authored-by: Carl <carl@solana.com>
2020-09-01 22:06:06 -07:00
carllin c8d67aa8eb
Add option for repairing only from trusted validators (#11752)
Co-authored-by: Carl <carl@solana.com>
2020-08-21 00:35:11 -07:00
carllin 7e25130529
Send votes from banking stage to vote listener (#11434)
*  Send votes from banking stage to vote listener

Co-authored-by: Carl <carl@solana.com>
2020-08-07 11:21:35 -07:00
carllin bf18524368
Add hook for getting vote transactions on replay (#11264)
* Add hook for getting vote transactions on replay

Co-authored-by: Carl <carl@solana.com>
2020-07-29 23:17:40 -07:00
carllin 73f3d04798
Add replay votes to gossip vote tracking (#11119)
* Plumb replay vote channel for notifying vote listener of replay votes

* Keep gossip only notification for debugging gossip in the future

Co-authored-by: Carl <carl@solana.com>
2020-07-20 17:29:07 -07:00
carllin 7a14e359d7
Switch to using weighted repair in RepairService (#10735)
* Plumb votes into repair service

* Remove refactoring

* Fix tests

* Switch to using RepairWeight for generating repairs

* Revert "Weight repair slots based on vote stake (#10741)"

This reverts commit cabd0a09c3.

* Update logging

Co-authored-by: Carl <carl@solana.com>
2020-07-10 05:52:54 +00:00
sakridge 58a475b789
Add db recovery methods (#10838) 2020-07-06 12:43:45 -07:00
Greg Fitzgerald 50b3fa83a0
Move BankCommitmentCache to solana_runtime (#10816)
* Remove Blockstore member variable from BlockCommitmentCache

* Hoist is_confirmed_rooted() to its only caller

BlockCommitmentCache no longer depends on Blockstore

* Move BlockCommitmentCache to solana_runtime
2020-06-25 22:06:58 -06:00
Ryo Onodera 4164c69683
Tiny cleanups in preparation for persistent tower (#10805) 2020-06-25 18:24:16 +09:00
sakridge cabd0a09c3
Weight repair slots based on vote stake (#10741)
* Weight repair slots based on vote stake

* Add test
2020-06-22 20:27:45 -07:00
Greg Fitzgerald 6ee222363e
Move BankForks to solana_runtime (#10637)
* Move BankForks to solana_runtime

* Update imports
2020-06-17 15:27:03 +00:00
Greg Fitzgerald 2eb6f498a8
Remove redundant BankForks parameter (#10537) 2020-06-12 11:04:17 -06:00
Michael Vines ad7b113944
Ignore test_tvu_exit (#10134)
automerge
2020-05-19 23:40:27 -07:00
Kristofer Peterson 58ef02f02b
9951 clippy errors in the test suite (#10030)
automerge
2020-05-15 09:35:43 -07:00
Jack May eb1acaf927
Remove archiver and storage program (#9992)
automerge
2020-05-14 18:22:47 -07:00
sakridge f98bfda6f9
Security changes (#9923)
* Move test-only functions to test modules

* Remove sigverify disable

* Remove chacha CTR code
2020-05-08 10:00:23 -07:00
Tyera Eulberg 754c65c066
Refactor RPC subscriptions account handling (#9888)
* Switch subscriptions to use commitment instead of confirmations

* Add bank method to return account and last-modified slot

* Add last_modified_slot to subscription data and use to filter account subscriptions

* Update tests to non-zero last_notified_slot

* Add accounts subscriptions to test; fails at higher tx load

* Pass BankForks to RpcSubscriptions

* Use BankForks on add_account_subscription to properly initialize last_notified_slot

* Bundle subscriptions

* Check for non-equality

* Use commitment to initialize last_notified_slot; revert context.slot chage
2020-05-07 00:23:06 -06:00
carllin 3442f36f8a
Repair alternate versions of dead slots (#9805)
Co-authored-by: Carl <carl@solana.com>
2020-05-05 14:07:21 -07:00
sakridge fa20963b93
Revert shred fs (#9712)
* Revert "Untar is called for shred archives that do not exist. (#9565)"

This reverts commit 729cb5eec6.

* Revert "Dont insert shred payload into rocksdb (#9366)"

This reverts commit 5ed39de8c5.
2020-04-24 15:04:23 -07:00
Tyera Eulberg d5abff82e0
Add largest_confirmed_root to BlockCommitmentCache (#9640)
* Add largest_confirmed_root to BlockCommitmentCache

* clippy

* Add blockstore to BlockCommitmentCache to check root

* Add rooted_stake helper fn and test

* Nodes that are behind should correctly id confirmed roots

* Simplify rooted_stake collector
2020-04-22 12:22:09 -06:00
carllin bab3502260
Push down cluster_info lock (#9594)
* Push down cluster_info lock

* Rework budget decrement

Co-authored-by: Carl <carl@solana.com>
2020-04-21 12:54:45 -07:00
Michael Vines ad186b8652
Flag test_tvu_exit as serial to hopefully reduce CI flakiness (#9509) 2020-04-21 12:54:16 -07:00
anatoly yakovenko 5ed39de8c5
Dont insert shred payload into rocksdb (#9366)
automerge
2020-04-16 18:20:55 -07:00
sakridge 66abe45ea1
Decouple accounts hash calculation from snapshot hash (#9507) 2020-04-16 15:12:20 -07:00
Ryo Onodera b28ec430e4
Introduce background stale AppendVec shrink mechanism (#9219)
* Introduce background AppendVec shrink mechanism

* Support ledger tool

* Clean up

* save

* save

* Fix CI

* More clean up

* Add tests

* Clean up yet more

* Use account.hash...

* Fix typo....

* Add comment

* Rename accounts_cleanup_service
2020-04-06 17:30:23 +09:00
sakridge a61ddb6f61
max_ledger_slots -> max_ledger_shreds (#9198)
automerge
2020-03-31 17:21:19 -07:00
Michael Vines 0e2722c638
solana-validator now supports multiple --authorized-voter arguments (#9174)
* Use Epoch type

* Vote account's authorized voter is now supported without a validator restart
2020-03-31 08:23:42 -07:00
carllin 66946a4680
Check ClusterSlots for confirmation of block propagation (#9115) 2020-03-30 19:57:11 -07:00
Tyera Eulberg 50fa577af8
Use cluster confirmations in rpc and pubsub (#9138)
* Add runtime methods to simply get status and slot

* Add helper function to get slot confirmation_count from BlockCommitmentCache

* Return cluster confirmations in getSignatureStatus

* Remove use of invalid get_signature_confirmation_status

* Remove unused methods

* Update pubsub to use cluster confirmations

* Fix test_check_signature_subscribe failure

* Refactor confirmations to read commitment cache only once

* Review comments

* Use bank, root from BlockCommitmentCache

* Update docs

* Add metric for block-commitment aggregations

Co-authored-by: Justin Starry <justin@solana.com>
2020-03-30 17:53:25 -06:00
carllin 076fef5e57
Update Cluster Slots to support multiple threads (#9071)
Co-authored-by: Carl <carl@solana.com>
2020-03-25 18:09:19 -07:00
sakridge 4b397d15b3
Accounts cleanup service and perf improvements (#8799)
* Use atomic for ref count instead of taking rwlock

* Accounts cleanup service

* Review comments
2020-03-23 08:50:23 -07:00
Michael Vines aa24181a53
Remove blockstream unix socket support. RPC or bust (#9004)
automerge
2020-03-21 20:17:11 -07:00
Michael Vines 18c1f0dfe9
Remove stub core/src/genesis_utils.rs (#8999) 2020-03-21 10:54:40 -07:00
sakridge 453f5ce8f2
Shred filter (#8975)
Thread bank_forks into shred fetch
2020-03-20 07:49:48 -07:00
carllin dc1db33ec9
Add Capabilities to Signal BroadcastStage to Retransmit (#8899) 2020-03-19 23:35:01 -07:00
sakridge dc347dd3d7
Add Accounts hash consistency halting (#8772)
* Accounts hash consistency halting

* Add option to inject account hash faults for testing.

Enable option in local cluster test to see that node halts.
2020-03-16 08:37:31 -07:00
carllin 9872430bd2
Add VoteTracker for tracking cluster's votes in gossip (#8327)
Track votes by slot in cluster_vote_listener
2020-03-09 22:03:09 -07:00
Michael Vines cb6848aa80
Publish initial snapshot hash in gossip on validator startup (#8679)
automerge
2020-03-05 22:52:31 -08:00
Tyera Eulberg ab361a8073
Rename KeypairUtil to Signer (#8360)
automerge
2020-02-20 13:28:55 -08:00
sakridge 1720fe6a46
Snapshot hash gossip changes (#8358) 2020-02-20 11:46:13 -08:00
Michael Vines c4fd81fc1c The getConfirmedBlock RPC API is now disabled by default
The --enable-rpc-get-confirmed-block flag allows validators to opt-in to
the higher disk usage and IOPS.
2020-02-11 22:24:08 -07:00
Michael Vines 72b11081a4 Report validator rewards in getConfirmedBlock JSON RPC 2020-02-11 17:25:45 -07:00
sakridge 7058287273
Consensus fix, don't consider threshold check if.. (#7948)
* Consensus fix, don't consider threshold check if

lockouts are not increased

* Change partition tests to wait for epoch with > lockout slots

* Use atomic bool to signal partition
2020-01-27 16:49:25 -08:00
Sunny Gleason 5cf090c896
feat: implement RPC notification queue (#7863) 2020-01-20 16:08:29 -05:00
Greg Fitzgerald b5dba77056 Rename blocktree to blockstore (#7757)
automerge
2020-01-13 13:13:52 -08:00