Commit Graph

293 Commits

Author SHA1 Message Date
Tyera Eulberg cbb8b79a60
Add validator flag to opt in to cpi and logs storage (#14922)
* Add validator flag to opt in to cpi and logs storage

* Default TestValidator to opt-in; allow using in multinode-demo

* No clone

Co-authored-by: Carl Lin <carl@solana.com>
2021-02-01 14:00:51 -07:00
Tyera Eulberg ffa5c7dcc8
Deprecate commitment variants (#14797)
* Deprecate commitment variants

* Add new CommitmentConfig builders

* Add helpers to avoid allowing deprecated variants

* Remove deprecated transaction-status code

* Include new commitment variants in runtime commitment; allow deprecated as long as old variants persist

* Remove deprecated banks code

* Remove deprecated variants in core; allow deprecated in rpc/rpc-subscriptions for now

* Heavier hand with rpc/rpc-subscription commitment

* Remove deprecated variants from local-cluster

* Remove deprecated variants from various tools

* Remove deprecated variants from validator

* Update docs

* Remove deprecated client code

* Add new variants to cli; remove deprecated variants as possible

* Don't send new commitment variants to old clusters

* Retain deprecated method in test_validator_saves_tower

* Fix clippy matches! suggestion for BPF solana-sdk legacy compile test

* Refactor node version check to handle commitment variants and transaction encoding

* Hide deprecated variants from cli help

* Add cli App comments
2021-01-26 19:23:07 +00:00
Michael Vines bf1943e489 Add solana-test-validator --warp-slot argument 2021-01-22 21:17:02 -08:00
carllin 6dfad0652f
Cache account stores, flush from AccountsBackgroundService (#13140) 2021-01-11 17:00:23 -08:00
Michael Vines a95675a7ce
Avoid tmp snapshot backlog in SnapshotPackagerService under high load (#14516) 2021-01-11 10:21:15 -08:00
Jeff Washington (jwash) 773b21b34e
consolidate DEFAULT_HASHES_PER_TICK (#14463) 2021-01-07 09:49:42 -06:00
carllin 5affd8aa72
Add secondary indexes (#14212) 2020-12-31 18:06:03 -08:00
Michael Vines fb6c660cfd Port ip-echo-server to tokio 0.3 2020-12-30 09:29:16 -08:00
Trent Nelson fe667db910 validator: Add experimental flag to select PoH pinned core 2020-12-29 19:15:44 -07:00
Trent Nelson 82f61c0c4a core: Update stale error message 2020-12-29 19:15:44 -07:00
sakridge 2074e407cd
Add poh speed check and tick speed calibration (#14292) 2020-12-29 09:35:57 -08:00
Michael Vines 9ddd6f08e8 Persist gossip contact info 2020-12-27 20:46:54 -08:00
behzad nouri 49019c6613
obtains staked-nodes from the root-bank (#14257)
... as opposed to the working bank
2020-12-27 13:28:05 +00:00
Michael Vines ace360ade2 Multiple entrypoint support 2020-12-22 18:35:31 -08:00
sakridge baa9602411
Add shrink paths (#14238) 2020-12-21 21:33:37 -08:00
behzad nouri 7b08cb1f0d
improves performance in replay-stage (#14217)
bank::vote_accounts returns a hash-map which is slow to iterate, but all uses
only require an iterator:
https://github.com/solana-labs/solana/blob/b3dc98856/runtime/src/bank.rs#L4300-L4306
Similarly, calculate_stake_weighted_timestamp takes a hash-map whereas it only
requires an iterator:
https://github.com/solana-labs/solana/blob/b3dc98856/sdk/src/stake_weighted_timestamp.rs#L21-L28
2020-12-21 19:18:19 +00:00
sakridge da7d1e2302
Improved Transaction Forwarding (#13944)
* Forwarding

* Dedupe leaders

* Use consistent commitment for last_valid_slot in rpc send_transaction

* Plumb rpc send_transaction options into solana-validator

* Extend num slots banking-stage holds forwarded txs

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-12-17 15:37:22 -07:00
Michael Vines 0b92720fdb Don't require increased open file limit in solana-test-validator
Travis CI in particular does not allow the open file limit to be
increased.
2020-12-16 20:43:08 -08:00
Michael Vines bbad3fe501 TestValidator now implements Drop, no need to close() it 2020-12-11 04:17:38 +00:00
Michael Vines 0a9ff1dc9d Initial solana-test-validator command-line program 2020-12-11 04:17:38 +00:00
Ryo Onodera 3d9d7557c8
core/validator: Wrap std::process:exit(1) for easier testing (#13990) 2020-12-07 16:43:03 +00:00
Alexander Meißner a706706572
Validator CLI option to enable just-in-time compilation of BPF (#13789)
* Adds a CLI option to the validator to enable just-in-time compilation of BPF.

* Refactoring to use bpf_loader_program instead of feature_set to pass JIT flag from the validator CLI to the executor.
2020-12-07 09:49:55 +01:00
Tyera Eulberg 10c81a2448
Remove rpc_banks from validator (#13882)
* Remove rpc_banks from validator

* Bump abi-digest
2020-12-02 03:25:09 +00:00
Michael Vines 3eece38ffa Add expects() to improve error logs on join failures 2020-12-01 17:58:28 -08:00
behzad nouri e1793e5a13
caches vote-state de-serialized from vote accounts (#13795)
Gossip and other places repeatedly de-serialize vote-state stored in
vote accounts. Ideally the first de-serialization should cache the
result.

This commit adds new VoteAccount type which lazily de-serializes
VoteState from Account data and caches the result internally.

Serialize and Deserialize traits are manually implemented to match
existing code. So, despite changes to frozen_abi, this commit should be
backward compatible.
2020-11-30 17:18:33 +00:00
sakridge c1eb350c47
Allow contact debug interval to be adjusted (#13737) 2020-11-20 14:47:37 -08:00
Michael Vines 5d72e52ad0 Disable the PubSub vote subscription by default
The --rpc-pubsub-enable-vote-subscription flag may be used to enable it.
The current vote subscription is problematic because it emits a
notification for *every* vote, so hundreds a second in a real cluster.
Critically it's also missing information about *who* is voting,
rendering all those notifications practically useless.

Until these two issues can be resolved, the vote subscription is not
much more than a potential DoS vector.
2020-11-14 12:36:37 -08:00
sakridge c1f3f9d27b
Stop searching for incorrect shred version after a minute (#13512) 2020-11-12 14:01:13 -08:00
carllin 9821a7754c
Discard pre hard fork persisted tower if hard-forking (#13536)
* Discard pre hard fork persisted tower if hard-forking

* Relax config.require_tower

* Add cluster test

* nits

* Remove unnecessary check

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-12 23:29:04 +09:00
Trent Nelson 38f15e41b5 Validator: Periodically log what we're waiting for during `--wait-for-supermajority` 2020-11-11 20:03:26 +00:00
sakridge c644b05c54
Fix avx check with newest nightly compiler (#13465) 2020-11-09 08:04:34 -08:00
Justin Starry 8b0242a5d8
Allow nodes to advertise a different rpc address over gossip (#13053)
* Allow nodes to advertise a different rpc address over gossip

* Feedback
2020-10-22 03:31:48 +00:00
Ryo Onodera 0776fa05c7
Add ledger-tool dead-slots and improve purge a lot (#13065)
* Add ledger-tool dead-slots and improve purge a lot

* Reduce batch size...

* Add --dead-slots-only and fixed purge ordering
2020-10-21 17:45:21 +00:00
Ryo Onodera efdb560e97
Various clean-ups before assert adjustment (#13006)
* Various clean-ups before assert adjustment

* oops
2020-10-21 10:26:20 +09:00
Trent Nelson 3b3f7341fa validator: Activate RPC before halting on slot 2020-10-20 02:09:07 +00:00
Ryo Onodera a44e4d386f
Better tower logs for SwitchForkDecision and etc (#12875)
* Better tower logs for SwitchForkDecision and etc

* nits

* Update comment
2020-10-15 18:30:33 +09:00
Ryo Onodera 1f4bcf70b0
Fix various ledger-tool error due to no builtins (#12759)
* Fix various ledger-tool error due to no builtins

* Add missing file...
2020-10-09 12:19:36 -06:00
Michael Vines f41a73d76a Expose validator cli arguments for pubsub buffer tuning 2020-10-01 20:30:40 -07:00
Tyera Eulberg 89621adca7
Rpc -> proper optimistic confirmation (#12514)
* Add service to track the most recent optimistically confirmed bank

* Plumb service into ClusterInfoVoteListener and ReplayStage

* Clean up test

* Use OptimisticallyConfirmedBank in RPC

* Remove superfluous notifications from RpcSubscriptions

* Use crossbeam to avoid mpsc recv_timeout panic

* Review comments

* Remove superfluous last_checked_slots, but pass in OptimisticallyConfirmedBank for complete correctness
2020-09-28 20:43:05 -06:00
carllin 06f84c65f1
Fix rooted accounts cleanup, simplify locking (#12194)
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-28 16:04:46 -07:00
Michael Vines 31696a1d72 Port BPFLoader2 activation to FeatureSet and rework built-in program activation 2020-09-28 12:50:19 -07:00
sakridge 68e5a2ef56
Add RPC notify and banking keys debug (#12396) 2020-09-23 18:46:42 -07:00
Josh 65a6bfad09
Add blockstore column to store performance sampling data (#12251)
* Add blockstore column to store performance sampling data

* introduce timer and write performance metrics to blockstore

* introduce getRecentPerformanceSamples rpc

* only run on rpc nodes enabled with transaction history

* add unit tests for get_recent_performance_samples

* remove RpcResponse from rpc call

* refactor to use Instant::now and elapsed for timer

* switch to root bank and ensure not negative subraction

* Add PerfSamples to purge/compaction

* refactor to use Instant::now and elapsed for timer

* switch to root bank and ensure not negative subraction

* remove duplicate constants

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-09-22 12:26:32 -07:00
Michael Vines 208dd1de3a Move TestValidator into its own module 2020-09-19 08:35:26 -07:00
Michael Vines 1a03afccb1 validator/ cleanup 2020-09-19 08:35:26 -07:00
Ryo Onodera cb8661bd49
Persistent tower (#10718)
* Save/restore Tower

* Avoid unwrap()

* Rebase cleanups

* Forcibly pass test

* Correct reconcilation of votes after validator resume

* d b g

* Add more tests

* fsync and fix test

* Add test

* Fix fmt

* Debug

* Fix tests...

* save

* Clarify error message and code cleaning around it

* Move most of code out of tower save hot codepath

* Proper comment for the lack of fsync on tower

* Clean up

* Clean up

* Simpler type alias

* Manage tower-restored ancestor slots without banks

* Add comment

* Extract long code blocks...

* Add comment

* Simplify returned tuple...

* Tweak too aggresive log

* Fix typo...

* Add test

* Update comment

* Improve test to require non-empty stray restored slots

* Measure tower save and dump all tower contents

* Log adjust and add threshold related assertions

* cleanup adjust

* Properly lower stray restored slots priority...

* Rust fmt

* Fix test....

* Clarify comments a bit and add TowerError::TooNew

* Further clean-up arround TowerError

* Truly create ancestors by excluding last vote slot

* Add comment for stray_restored_slots

* Add comment for stray_restored_slots

* Use BTreeSet

* Consider root_slot into post-replay adjustment

* Tweak logging

* Add test for stray_restored_ancestors

* Reorder some code

* Better names for unit tests

* Add frozen_abi to SavedTower

* Fold long lines

* Tweak stray ancestors and too old slot history

* Re-adjust error conditon of too old slot history

* Test normal ancestors is checked before stray ones

* Fix conflict, update tests, adjust behavior a bit

* Fix test

* Address review comments

* Last touch!

* Immediately after creating cleaning pr

* Revert stray slots

* Revert comment...

* Report error as metrics

* Revert not to panic! and ignore unfixable test...

* Normalize lockouts.root_slot more strictly

* Add comments for panic! and more assertions

* Proper initialize root without vote account

* Clarify code and comments based on review feedback

* Fix rebase

* Further simplify based on assured tower root

* Reorder code for more readability

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-19 14:03:54 +09:00
Michael Vines daae638781 Add --gossip-validator argument 2020-09-14 20:18:27 -07:00
Michael Vines 6f325d4594 `solana-validator --rpc-bind-address` argument now works as expected 2020-09-10 13:36:13 -07:00
Tyera Eulberg 05db41fe9c
Cache block time in Blockstore (#11955)
* Add blockstore column to cache block times

* Add method to cache block time

* Add service to cache block time

* Update rpc getBlockTime to use new method, and refactor blockstore slightly

* Return block_time with confirmed block, if available

* Add measure and warning to cache-block-time
2020-09-09 09:33:14 -06:00
anatoly yakovenko c67f8bd821
Forward transactions to the expected leader instead of your own TPU port (#12004)
* Use PoHRecorder to send to the right leader

* cleanup

* fmt

* clippy

* Cleanup, fix bug

Co-authored-by: Carl <carl@solana.com>
2020-09-08 17:00:49 +08:00
carllin 1c1a3f979d
Detect and notify when deserializable shreds are available (#11816)
* Add logic to check for complete data ranges

* Add RPC signature notification

Co-authored-by: Carl <carl@solana.com>
2020-09-01 22:06:06 -07:00
carllin c8d67aa8eb
Add option for repairing only from trusted validators (#11752)
Co-authored-by: Carl <carl@solana.com>
2020-08-21 00:35:11 -07:00
carllin b7ed06b17a
Cleanup test utilities (#11723)
* Add voting utility

* Add blockstore utility

Co-authored-by: Carl <carl@solana.com>
2020-08-20 05:04:38 +00:00
Jack May 7c736f71fe
Make BPF Loader static (#11516) 2020-08-14 12:32:45 -07:00
Jack May 9290e561e1
Align host addresses (#11384)
* Align host addresses

* support new program abi

* update epoch rollout

* Enforce aligned pointers in cross-program invocations
2020-08-11 16:11:52 -07:00
Greg Fitzgerald edadd5d6d5
Remove Budget from CLI (#11451)
* Remove support for Budget

Also:
* Make "pay" command a deprecated alias for the "transfer" command

* chore: remove budget from web3.js

* Drop Budget depedency from core

Validators no longer ship with builtin Budget
2020-08-07 16:01:51 -06:00
carllin 7e25130529
Send votes from banking stage to vote listener (#11434)
*  Send votes from banking stage to vote listener

Co-authored-by: Carl <carl@solana.com>
2020-08-07 11:21:35 -07:00
Greg Fitzgerald bad486823c
Add a client for BankForks (#10728)
Also:
* Use BanksClient in solana-tokens
2020-08-07 08:45:17 -06:00
carllin bf18524368
Add hook for getting vote transactions on replay (#11264)
* Add hook for getting vote transactions on replay

Co-authored-by: Carl <carl@solana.com>
2020-07-29 23:17:40 -07:00
carllin 6578ad7d08
Speed up local cluster partitioning tests (#11177)
* Fix long local cluster partition tests by skipping slot warmup

Co-authored-by: Carl <carl@solana.com>
2020-07-23 18:50:42 -07:00
Tyera Eulberg b5a6a2f461
Initialize BlockCommitmentCache slot and root on node boot (#11178)
* Initialize commitment-cache slot and root on node boot

* Ignore long tests
2020-07-23 11:44:57 -06:00
carllin 73f3d04798
Add replay votes to gossip vote tracking (#11119)
* Plumb replay vote channel for notifying vote listener of replay votes

* Keep gossip only notification for debugging gossip in the future

Co-authored-by: Carl <carl@solana.com>
2020-07-20 17:29:07 -07:00
sakridge 7fe870ba48
Make accounts file clean faster (#11071) 2020-07-15 09:37:40 -07:00
Tyera Eulberg 9a80e31bae
Expose tss to the other blockstore_processor path (#11070) 2020-07-14 21:14:48 -06:00
carllin 7a14e359d7
Switch to using weighted repair in RepairService (#10735)
* Plumb votes into repair service

* Remove refactoring

* Fix tests

* Switch to using RepairWeight for generating repairs

* Revert "Weight repair slots based on vote stake (#10741)"

This reverts commit cabd0a09c3.

* Update logging

Co-authored-by: Carl <carl@solana.com>
2020-07-10 05:52:54 +00:00
Greg Fitzgerald 2fdbb97244
Rename largest_confirmed_root to highest_confirmed_root (#10947) 2020-07-07 23:59:46 +00:00
sakridge d492f4f15b
Clean accounts path on validator startup (#10929) 2020-07-07 09:41:45 -07:00
sakridge 58a475b789
Add db recovery methods (#10838) 2020-07-06 12:43:45 -07:00
Ryo Onodera ecf6fb2cd2
Rename new_banks_from_blockstore => new_banks_from_ledger (#10864) 2020-07-01 06:19:40 +00:00
sakridge 1ffad2d051
More guard rails for restart with ledger procedure (#10853)
* Add expected_bank_hash required for supermajority
* Print snapshot hash in ledger-tool create-snapshot.
2020-06-30 12:43:48 -07:00
sakridge 583cec922b
Remove ledger purge batching (#10830) 2020-06-29 14:44:35 -07:00
Greg Fitzgerald 50b3fa83a0
Move BankCommitmentCache to solana_runtime (#10816)
* Remove Blockstore member variable from BlockCommitmentCache

* Hoist is_confirmed_rooted() to its only caller

BlockCommitmentCache no longer depends on Blockstore

* Move BlockCommitmentCache to solana_runtime
2020-06-25 22:06:58 -06:00
Ryo Onodera 4164c69683
Tiny cleanups in preparation for persistent tower (#10805) 2020-06-25 18:24:16 +09:00
sakridge a1ef921b88
Rework backup and clear function (#10751) 2020-06-23 21:29:07 +00:00
sakridge 2ba8fc5243
Remove slots past wait-for-supermajority slot. (#10720) 2020-06-22 20:27:25 -07:00
Greg Fitzgerald 0550b893b0
Fix typos (#10675)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-06-17 20:54:52 -07:00
Greg Fitzgerald 6ee222363e
Move BankForks to solana_runtime (#10637)
* Move BankForks to solana_runtime

* Update imports
2020-06-17 15:27:03 +00:00
sakridge 0de6c444d6
Simd poh (#10604)
* Simd poh

* Fix poh verify bench
2020-06-16 23:03:26 -07:00
Kristofer Peterson e23340d89e
Clippy cleanup for all targets and nighly rust (also support 1.44.0) (#10445)
* address warnings from 'rustup run beta cargo clippy --workspace'

minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs

expect some performance improvement AccountsDB::clean_accounts()

* address warnings from 'rustup run beta cargo clippy --workspace --tests'

* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'

* rustfmt

* fix warning stragglers

* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned

* Remove lock around JsonRpcRequestProcessor (#10417)

automerge

* make ancestors parameter optional to avoid forcing construction of empty hash maps

Co-authored-by: Greg Fitzgerald <greg@solana.com>
2020-06-09 09:38:14 +09:00
Michael Vines 0442c45d5b
Feign RPC health while in a `--wait-for-supermajority` holding pattern (#10295) 2020-05-28 12:22:19 -07:00
Michael Vines 486168b796
Revert "Add AVX2 runtime checks (#10033)" (#10167)
This reverts commit cf8eb7700b.
2020-05-21 13:13:52 -07:00
Tyera Eulberg bac4aec16f
Trigger RPC notifications after block commitment cache update (#10077)
* Fixup commitment-aggregation metric

* Trigger notifications after commitment-cache update

* Fixup fn name

* Add single-confirmation commitment level

* Rename to highest_confirmed_slot

* Pass commitment-cache info directly to notifications

* Use match

* Update commitment docs

* Update out of date pubsub docs
2020-05-18 12:49:01 -06:00
Reisen bfcfbab818
Add Vote PubSub endpoint for live gossip votes. (#10045)
* Add Vote PubSub endpoint for live gossip votes.

* Updated tests for Vote RPC and Vote Listener

* Add JSON RPC documentation for Vote RPC.

* Base58 encode hash in Vote RPC response.
2020-05-17 15:01:08 -06:00
Michael Vines 1da1667920
Forge a confirmed root before halting for RPC inspection (#10061) 2020-05-15 09:02:48 -07:00
Jack May eb1acaf927
Remove archiver and storage program (#9992)
automerge
2020-05-14 18:22:47 -07:00
Trent Nelson cf8eb7700b
Add AVX2 runtime checks (#10033)
automerge
2020-05-13 12:19:22 -07:00
sakridge f98bfda6f9
Security changes (#9923)
* Move test-only functions to test modules

* Remove sigverify disable

* Remove chacha CTR code
2020-05-08 10:00:23 -07:00
Tyera Eulberg 754c65c066
Refactor RPC subscriptions account handling (#9888)
* Switch subscriptions to use commitment instead of confirmations

* Add bank method to return account and last-modified slot

* Add last_modified_slot to subscription data and use to filter account subscriptions

* Update tests to non-zero last_notified_slot

* Add accounts subscriptions to test; fails at higher tx load

* Pass BankForks to RpcSubscriptions

* Use BankForks on add_account_subscription to properly initialize last_notified_slot

* Bundle subscriptions

* Check for non-equality

* Use commitment to initialize last_notified_slot; revert context.slot chage
2020-05-07 00:23:06 -06:00
Michael Vines 09ae61651a
Eliminate BankForksInfo (#9887) 2020-05-06 08:24:59 -07:00
Ryo Onodera a91236012d
Pass around --max-genesis-archive-unpacked-size (#9161)
automerge
2020-04-29 18:53:34 -07:00
Michael Vines 230df0ec0c
Upgrade to Rust 1.43.0 (#9754) 2020-04-29 18:02:05 -07:00
sakridge fa20963b93
Revert shred fs (#9712)
* Revert "Untar is called for shred archives that do not exist. (#9565)"

This reverts commit 729cb5eec6.

* Revert "Dont insert shred payload into rocksdb (#9366)"

This reverts commit 5ed39de8c5.
2020-04-24 15:04:23 -07:00
Tyera Eulberg d5abff82e0
Add largest_confirmed_root to BlockCommitmentCache (#9640)
* Add largest_confirmed_root to BlockCommitmentCache

* clippy

* Add blockstore to BlockCommitmentCache to check root

* Add rooted_stake helper fn and test

* Nodes that are behind should correctly id confirmed roots

* Simplify rooted_stake collector
2020-04-22 12:22:09 -06:00
carllin bab3502260
Push down cluster_info lock (#9594)
* Push down cluster_info lock

* Rework budget decrement

Co-authored-by: Carl <carl@solana.com>
2020-04-21 12:54:45 -07:00
Michael Vines a7aa7e172b
validator: Consider the activated stake of this node to be online again (#9573)
automerge
2020-04-18 10:16:19 -07:00
Michael Vines 8509dcb8a0 Report offline/wrong-shred nodes while waiting for a super majority in gossip 2020-04-17 13:32:19 -07:00
anatoly yakovenko 5ed39de8c5
Dont insert shred payload into rocksdb (#9366)
automerge
2020-04-16 18:20:55 -07:00
sakridge 66abe45ea1
Decouple accounts hash calculation from snapshot hash (#9507) 2020-04-16 15:12:20 -07:00
Jack May 241a05fc52
Add native loader entry points (#9486) 2020-04-15 09:41:29 -07:00
Michael Vines 40737e9efa Add health check URI 2020-04-15 09:06:24 -07:00