Justin Starry
a7e65c0034
RPC: use finalized as default pubsub commitment level ( #16659 )
...
* RPC: use finalized as default pubsub commitment level
* update docs
* Fix tests
2021-04-20 08:19:54 +00:00
Michael Vines
c8b474cd0b
Send votes to next leader's TPU instead of our TPU
2021-04-20 00:38:21 -07:00
Jeff Washington (jwash)
4aa753ff01
rename threads: 15 char limit ( #16625 )
2021-04-19 12:16:58 -05:00
Michael Vines
b06e93fe5b
Increase test timeout
2021-04-18 20:55:02 -07:00
Michael Vines
a911ae00ba
clippy
2021-04-18 20:55:02 -07:00
behzad nouri
e405747409
Revert "Add limit and shrink policy for recycler ( #15320 )"
...
This reverts commit c2e8814dce
.
2021-04-18 19:29:24 +00:00
Michael Vines
6907a2366e
Remove unnecessary clone
2021-04-17 10:23:13 -07:00
steviez
bb24318ef0
Document shreds ( #16514 )
...
No functionality changes from this commit
2021-04-16 14:04:46 -05:00
Tyera Eulberg
974e6dd2c1
Deprecate "confirmed" RpcClient methods ( #16520 )
...
* Remove obsolete client methods
* Deprecate GetConfirmed client methods
* Rename Confirmed config structs, with appropriate deprecation
* Fixup client apps
* Map RpcRequest to deprecated when targeting older nodes
2021-04-15 17:00:14 -06:00
Tyera Eulberg
7dfb51c0b4
Cli: move airdrop to rpc requests ( #16557 )
...
* Add recent_blockhash to requestAirdrop
* Move tx confirmation to separate method
* Add RpcClient airdrop methods
* Request cli airdrop via RpcClient
* Pass optional faucet_addr into TestValidator and fix tests
* Update client/src/rpc_client.rs
Co-authored-by: Michael Vines <mvines@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-04-15 06:25:23 +00:00
behzad nouri
d92721aab9
uses timeouts based on stake for filtering pull responses ( #16549 )
...
filter_pull_responses is using default timeout when discarding pull
responses (except for ContactInfo):
https://github.com/solana-labs/solana/blob/f804ce63c/core/src/crds_gossip_pull.rs#L349-L350
But purging code uses timeouts based on stake:
https://github.com/solana-labs/solana/blob/f804ce63c/core/src/cluster_info.rs#L1867-L1870
So the crds value will not be purged from the sender's table and will be
sent again over the next pull request.
2021-04-14 20:18:00 +00:00
behzad nouri
f35a6a8be0
prioritizes contact-infos in pull responses ( #16541 )
...
Expired crds values where the contact-info does not exist are wasted:
https://github.com/solana-labs/solana/blob/f804ce63c/core/src/crds_gossip_pull.rs#L353-L378
and then are sent again over the next pull-request.
Also, the stake of the first response (which can be anything) is used to
weight all pull-responses to a node, while the rest of responses can
have different stake.
https://github.com/solana-labs/solana/blob/f804ce63c/core/src/cluster_info.rs#L2231
2021-04-14 18:45:20 +00:00
carllin
f0c150cfb9
Fix channel panic in tests ( #16503 )
...
* Fix channel panic
* Add exit signal to PohRecorder because Crossbeam doesnt drop objects inside dropped channel
2021-04-14 12:07:21 -05:00
Justin Starry
f641429056
Remove blake3 from bpf program dependencies ( #16506 )
2021-04-13 10:06:05 +00:00
Tyera Eulberg
37afa00ffb
Rpc: deprecate getConfirmed endpoints ( #16502 )
...
* Deprecate getConfirmed methods in rpc
* Add new methods to docs
* Move deprecated rpc methods to separate docs section
* Add note to docs about removal timing
2021-04-13 01:50:15 -06:00
Justin Starry
85eb37fab0
Merge pull request from GHSA-8v47-8c53-wwrc
...
* Track transaction check time separately from account loads
* banking packet process metrics
* Remove signature clone in status cache lookup
* Reduce allocations when converting packets to transactions
* Add blake3 hash of transaction messages in status cache
* Bug fixes
* fix tests and run fmt
* Address feedback
* fix simd tx entry verification
* Fix rebase
* Feedback
* clean up
* Add tests
* Remove feature switch and fall back to signature check
* Bump programs/bpf Cargo.lock
* clippy
* nudge benches
* Bump `BankSlotDelta` frozen ABI hash`
* Add blake3 to sdk/programs/Cargo.lock
* nudge bpf tests
* short circuit status cache checks
Co-authored-by: Trent Nelson <trent@solana.com>
2021-04-13 00:28:08 -06:00
Tyera Eulberg
70f3f7e679
Move obsolete rpc endpoints to separate api for removal ( #16500 )
...
* Move obsolete rpc methods to separate api for removal
* Remove obsolete method from docs
* Fix test using obs method
2021-04-12 20:33:40 -06:00
Michael Vines
2229b70c4e
Add authorized-voter add/remove-all commands
2021-04-12 15:55:28 -07:00
Michael Vines
17a173ebb5
Clean up build warning
2021-04-12 15:55:28 -07:00
carllin
dc7030ffaa
Allow fork choice to support multiple versions of a slot ( #16266 )
2021-04-12 01:00:59 -07:00
carllin
99b3aab703
Track gossip vote updates per hash for replay stage ( #16421 )
...
* Track gossip vote updates per hash for replay stage
2021-04-10 17:34:45 -07:00
Christian Drappi
54a04bac3d
Apple M1 compatibility ( #16346 )
...
Co-authored-by: Christian Drappi <christiandrappi@Christians-MacBook-Pro.local>
2021-04-09 17:21:01 -07:00
Tyera Eulberg
8bc0bdd40b
Fill in not-yet-finalized block-time if possible ( #16460 )
2021-04-09 20:25:47 +00:00
behzad nouri
22a18a68e3
stops consuming pinned vectors with a recycler ( #16441 )
...
If the vector is pinned and has a recycler, From<PinnedVec>
implementation of Vec should clone (instead of consuming) the underlying
vector so that the next allocation of a PinnedVec will recycle an
already pinned one.
2021-04-09 16:55:24 +00:00
François Garillot
b08cff9e77
Simplify some pattern-matches ( #16402 )
...
When those match an exact combinator on Option / Result.
Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust ).
2021-04-08 12:40:37 -06:00
Tyera Eulberg
bb9d2fd07a
Cli: use get_inflation_rewards and limit epochs queried ( #16408 )
...
* Fix block-with-limit when not finalized blocks found
* Enable confirmed commitment in getInflationReward
* Use get_inflation_rewards in cli
* Line up rewards output
* Add range validator
* Change cli epoch arg -> num epochs
* Add solana inflation rewards subcommand
* Consolidate epoch rewards meta
2021-04-08 10:57:33 -06:00
Josh
e501fa5f0b
Rpc: introduce get_inflation_reward rpc call ( #16278 )
...
* feat: introduce get_inflation_reward rpc call
* fix: style suggestions
* fix: more style changes and match how other rpc functions are defined
* feat: get reward for a single epoch
* feat: default to the most recent epoch
* fix: don't factor out get_confirmed_block
* style: introduce from impl for RpcEncodingConfigWrapper
* style: bring commitment into variable
* feat: support multiple pubkeys for get_inflation_reward
* feat: add get_inflation_reward to rpc client
* feat: return rewards in order
* fix: rename pubkeys to addresses
* docs: introduce jsonrpc docs for get_inflation_reward
* style: early return in map (not sure which is more idiomatic)
* fix: call the rpc client function args addresses as well
* fix: style
* fix: filter out only addresses we care about
* style: make this more idiomatic
* fix: change rpc client epoch to optional and include some docs edits
* feat: filter out rent rewards in get_inflation_reward
* feat: add option epoch config param to get_inflation_reward
* feat: rpc client get_inflation_reward takes epoch instead of config and some filter staking and voting rewards
2021-04-06 18:10:53 -07:00
carllin
1219842a96
No wallclock throttle tests ( #16396 )
2021-04-05 19:40:16 -07:00
Trent Nelson
b71875df61
cluster-info: Get rid of some integer math while we're here
2021-04-06 00:09:37 +00:00
Trent Nelson
b6b08706b9
cluster-info: Don't subtract non-shred spies from node count
2021-04-06 00:09:37 +00:00
Trent Nelson
7a2a39093d
validator: Use a const for wait for supermajority threshold
2021-04-05 17:29:37 -06:00
Michael Vines
43feef7362
Adjust tokio version to just "1"
2021-04-05 17:13:20 +00:00
Michael Vines
b242f82696
Reduce test-validator ledger size
2021-04-05 08:37:29 -07:00
behzad nouri
701fc93343
patches bug in banking stage where buffered packets are never retained ( #16276 )
...
banking_stage::handle_forwarding is retaining buffered packets with
empty index, so nothing is held:
https://github.com/solana-labs/solana/blob/6f3926b64/core/src/banking_stage.rs#L520
2021-04-05 12:46:21 +00:00
sakridge
3429785d9b
Wait for 90 percent of stake before starting ( #16340 )
2021-04-03 14:21:20 -07:00
carllin
4e5ef6bce2
Add cluster state verifier logging ( #16330 )
...
* Add cluster state verifier logging
* Add duplicate-slots iterator to ledger tool
2021-04-02 21:48:44 -07:00
Tyera Eulberg
da27acabcc
Rpc: enable getConfirmedSignaturesForAddress2 to return confirmed (not yet finalized) data ( #16281 )
...
* Update blockstore method to allow return of unfinalized signature
* Support confirmed sigs in getConfirmedSignaturesForAddress2
* Add deprecated comments
* Update docs
* Enable confirmed transaction-history in cli
* Return real confirmation_status; fill in not-yet-finalized block time if possible
2021-04-01 04:35:57 +00:00
Tyera Eulberg
18bd47dbe1
Rpc: fix getConfirmedTransaction slot ( #16288 )
...
* Fix transaction blockstore apis
* Update blockstore apis in rpc
2021-03-31 21:04:00 -06:00
behzad nouri
3f63ed9a72
removes OrderedIterator and transaction batch iteration order ( #16153 )
...
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817
However load_and_execute_transactions is iterating over
lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.
If iteratorion_order is [1, 2, 0], and i is 0, then:
lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).
This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.
2021-03-31 23:59:19 +00:00
sakridge
54c68ea83f
Drop write lock on sysvars ( #15497 )
...
* Drop write lock on sysvars
* adds env var for demoting sysvar write lock demotion
* moves demote logic to is_writable
* feature gates sysvar write lock demotion
* adds builtins to write lock demotion
* adds system program id to builtins
* adds Feature111...
* adds an abi-freeze test
* mvines set of builtin program keys
Co-authored-by: Michael Vines <mvines@gmail.com>
* update tests
* adds bpf loader keys
* Add test sysvar
* Plumb demote_sysvar to is_writable
* more plumbing of demote_sysvar_write_locks to is_writable
* patches test_program_bpf_instruction_introspection
* hard codes demote_sysvar_write_locks to false for serialization/encoding methods
* Revert "hard codes demote_sysvar_write_locks to false for serialization/encoding methods"
This reverts commit ae3e2d2e777437bddd753933097a210dcbc1b1fc.
* change the hardcoded ones to demote_sysvar_write_locks=true
* Use data_as_mut_slice
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-03-30 10:05:09 -07:00
Jeff Washington (jwash)
414c7070cb
poll checking for new record in poh service after every batch of hashes instead of busy waiting ( #16167 )
...
* poll waiting in poh service after every batch of hashes
* clippy
2021-03-30 10:34:21 -05:00
Jeff Washington (jwash)
5eff23db0c
TransactionRecorder uses unique channel so we can use Recv instead of RecvTimeout ( #16195 )
...
* time
* new channel each call
* new channel every time
2021-03-30 00:51:35 -05:00
Tyera Eulberg
3977ed5c82
Future-aware enum name
2021-03-29 14:58:05 -06:00
Tyera Eulberg
60ed8e2892
Rpc: enable getConfirmedBlocks and getConfirmedBlocksWithLimit to return confirmed (not yet finalized) data ( #16161 )
...
* Add commitment config capabilities
* Use rpc limit if no end_slot provided
* Limit to actually finalized blocks
* Support confirmed blocks in getConfirmedBlocks and getConfirmedBlocksWithLimit
* Update docs
* Add client plumbing
* Rename config enum
2021-03-29 12:41:31 -06:00
sakridge
60b4771fc6
Only print skipped leader slot message when the node is actually leader ( #16156 )
...
Also, check vote signature after the vote is signed
2021-03-26 17:45:53 -07:00
Tyera Eulberg
433f1ead1c
Rpc: enable getConfirmedBlock and getConfirmedTransaction to return confirmed (not yet finalized) data ( #16142 )
...
* Add Blockstore block and tx apis that allow unrooted responses
* Add TransactionStatusMessage, and send on bank freeze; also refactor TransactionStatusSender
* Track highest slot with tx-status writes complete
* Rename and unpub fn
* Add commitment to GetConfirmed input configs
* Support confirmed blocks in getConfirmedBlock
* Support confirmed txs in getConfirmedTransaction
* Update sigs-for-addr2 comment
* Enable confirmed block in cli
* Enable confirmed transaction in cli
* Review comments
* Rename blockstore method
2021-03-26 16:47:35 -06:00
Jeff Washington (jwash)
4f4cffbd03
Throttle PoH ticks by cumulative slot time ( #16139 )
...
* Throttle PoH ticks by cumulative slot time
* respond to pr feedback
* saturating sub
* updated comment
2021-03-26 18:54:16 +00:00
Jeff Washington (jwash)
06ac0fe9a3
increase timeout in TransactionRecorder.record ( #16133 )
2021-03-25 21:31:07 -05:00
sakridge
b99ae8f334
Skip leader slots until a vote lands ( #15607 )
2021-03-25 18:54:51 -07:00
behzad nouri
b041b55028
makes test_pull_request_time_pruning smaller ( #16128 )
2021-03-25 22:44:43 +00:00
sakridge
9b94741290
Fix test_replay_commitment_cache ( #16131 )
2021-03-25 21:16:39 +00:00
Justin Starry
e817a6db00
Add timeout for local cluster partition tests ( #16123 )
...
* Add timeout for local cluster partition tests
* fix optimistic conf test logs
* Bump instruction count assertions
2021-03-25 13:27:07 -06:00
carllin
52703badfa
Setup ReplayStage confirmation scaffolding for duplicate slots ( #9698 )
2021-03-24 23:41:52 -07:00
Tyera Eulberg
a8ef29df27
Support getBlockTime for unfinalized blocks ( #16103 )
2021-03-24 20:52:08 -06:00
sakridge
96ccc40f0a
Set ticks_per_slot higher for banking stage tests ( #16094 )
...
Tests are timing out because the bank hit the MaxTickHeight and
will not process the transactions.
2021-03-24 14:05:43 -07:00
Jeff Washington (jwash)
f68860a643
poh record metrics ( #16092 )
2021-03-24 14:48:32 -05:00
behzad nouri
a6c23648cb
limits CrdsGossipPull::pull_request_time size ( #15793 )
...
There is no pruning logic on CrdsGossipPull::pull_request_time
https://github.com/solana-labs/solana/blob/79ac1997d/core/src/crds_gossip_pull.rs#L172-L174
potentially allowing this to take too much memory.
Additionally, CrdsGossipPush::last_pushed_to is pruning recent push
timestamps:
https://github.com/solana-labs/solana/blob/79ac1997d/core/src/crds_gossip_push.rs#L275-L279
instead of the older ones.
Co-authored-by: Nathan Hawkins <utsl@utsl.org>
2021-03-24 18:33:56 +00:00
behzad nouri
570fd3f810
makes turbine peer computation consistent between broadcast and retransmit ( #14910 )
...
get_broadcast_peers is using tvu_peers:
https://github.com/solana-labs/solana/blob/84e52b606/core/src/broadcast_stage.rs#L362-L370
which is potentially inconsistent with retransmit_peers:
https://github.com/solana-labs/solana/blob/84e52b606/core/src/cluster_info.rs#L1332-L1345
Also, the leader does not include its own contact-info when broadcasting
shreds:
https://github.com/solana-labs/solana/blob/84e52b606/core/src/cluster_info.rs#L1324
but on the retransmit side, slot leader is removed only _after_ neighbors and
children are computed:
https://github.com/solana-labs/solana/blob/84e52b606/core/src/retransmit_stage.rs#L383-L384
So the turbine broadcast tree is different between the two stages.
This commit:
* Removes retransmit_peers. Broadcast and retransmit stages will use tvu_peers
consistently.
* Retransmit stage removes slot leader _before_ computing children and
neighbors.
2021-03-24 13:34:48 +00:00
Justin Starry
e7fd7d46cf
rpc: add getSlotLeaders method ( #16057 )
2021-03-23 17:48:54 +00:00
behzad nouri
4f82b897bc
buffers data shreds to make larger erasure coded sets ( #15849 )
...
Broadcast stage batches up to 8 entries:
https://github.com/solana-labs/solana/blob/79280b304/core/src/broadcast_stage/broadcast_utils.rs#L26-L29
which will be serialized into some number of shreds and chunked into FEC
sets of at most 32 shreds each:
https://github.com/solana-labs/solana/blob/79280b304/ledger/src/shred.rs#L576-L597
So depending on the size of entries, FEC sets can be small, which may
aggravate loss rate.
For example 16 FEC sets of 2:2 data/code shreds each have higher loss
rate than one 32:32 set.
This commit broadcasts data shreds immediately, but also buffers them
until it has a batch of 32 data shreds, at which point 32 coding shreds
are generated and broadcasted.
2021-03-23 14:52:38 +00:00
Jeff Washington (jwash)
57ba86c821
eliminate lock on record ( #15929 )
...
* eliminate lock on record
* use same error as MaxHeightReached
* clippy
* review feedback
* refactor should_tick code
* pr feedback
2021-03-23 09:10:04 -05:00
Jeff Washington (jwash)
2fc609a294
add metric for ticks from poh_recorder.record ( #16047 )
2021-03-22 15:35:06 -05:00
Tyera Eulberg
2ec24d438f
Make getStakeActivation response consistent for undelegated accounts ( #16038 )
2021-03-19 14:54:56 -06:00
Jeff Washington (jwash)
ddc758439e
metrics for poh_recorder.record ( #15998 )
2021-03-19 09:48:55 -05:00
Michael Vines
02b81dd05d
Update to reqwest 0.11.2
2021-03-18 11:59:41 -07:00
Tyera Eulberg
aa54c468ea
rpc: Add config options limiting getConfirmedBlock response data ( #15970 )
...
* Add new confirmed block struct
* Add RpcConfirmedBlockConfig options
* Configure block response based on new options
* Add client api, use in cli fetch_epoch_rewards
* Update docs
* Apply review suggestions
2021-03-18 17:58:20 +00:00
Michael Vines
04c99cf7ea
Add --slots-per-epoch argument
2021-03-17 22:56:41 +00:00
carllin
f548a04fae
Allow unbounded wallclock processing time in tests ( #15961 )
2021-03-17 15:48:50 -07:00
Michael Vines
59c19d9fbf
Notice the user when the --mint, --bpf-program, or --clone arguments are ignored
2021-03-17 20:04:53 +00:00
Michael Vines
8a9b51952e
Ignore flaky test_banking_stage_entries_only and test_banking_stage_entryfication
2021-03-17 11:28:56 -07:00
Jeff Washington (jwash)
40997d0aef
add metrics for tick producer and poh_recorder ( #15931 )
2021-03-17 10:38:26 -05:00
Jeff Washington (jwash)
5460fb10a2
drop poh lock after record ( #15930 )
2021-03-17 10:37:20 -05:00
Michael Vines
a2eb655322
=1.7.0
2021-03-16 07:51:07 +00:00
Jeff Washington (jwash)
efee8b62d7
a few missed set_data calls ( #15846 )
...
* a few missed set_data calls
* another set data call
2021-03-15 21:57:23 -05:00
Jeff Washington (jwash)
c09ea2c314
More AccountSharedData construction ( #15844 )
...
* one more AccountSharedData construction
* one more construct
2021-03-15 19:27:17 -05:00
carllin
c1ba265dd9
Wallclock BankingStage Throttle ( #15731 )
2021-03-15 17:11:15 -07:00
Michael Vines
0c9ca5522c
Bump version to v1.7.0
2021-03-13 09:01:21 +00:00
Tyera Eulberg
5b2da19c93
Rpc: support extended config for getConfirmedBlock ( #15827 )
...
* Add rpc confirmed-block config wrapper to support struct of extended config
* Update docs
* Make config wrapper generic and use in getConfirmedTransaction as well
* Update/clean confirmed-tx docs
2021-03-12 22:19:45 +00:00
behzad nouri
f2865dfd63
requires stakes for propagating crds values through gossip ( #15561 )
2021-03-12 15:50:14 +00:00
Justin Starry
918d04e3f0
Add more slot update notifications ( #15734 )
...
* Add more slot update notifications
* fix merge
* Address feedback and add integration test
* switch to datapoint
* remove unused shred method
* fix clippy
* new thread for rpc completed slots
* remove extra constant
* fixes
* rely on channel closing
* fix check
2021-03-12 21:44:06 +08:00
Ryo Onodera
4bbeb9c033
Remove old feature: simple_capitalization ( #15763 )
...
* Remove old feature: simple_capitalization
* Fix another failing test in core
* Finish up test cleanup
* Further clean up a bit
2021-03-12 11:12:40 +09:00
Jeff Washington (jwash)
952c3bcbb7
AccountSharedData construction ( #15790 )
2021-03-11 18:09:04 -06:00
Trent Nelson
24d18b3cf2
docs: add docs links for crates published to crates.io
2021-03-11 08:38:18 +00:00
Jeff Washington (jwash)
1135ffd595
mut data refs as slice ( #15782 )
2021-03-10 15:28:03 -06:00
behzad nouri
56923c91bf
limits number of unique pubkeys in the crds table ( #15539 )
2021-03-10 20:46:05 +00:00
Jeff Washington (jwash)
52e54e1100
account.data -> data() ( #15778 )
2021-03-09 22:31:33 +00:00
Jeff Washington (jwash)
8a3135d17b
Account->AccountSharedData ( #15691 )
2021-03-09 15:06:07 -06:00
carllin
2bee9435f3
Add tracer key for tracing transaction path through the network ( #15732 )
2021-03-08 19:31:00 -08:00
carllin
331c45decf
Report datapoint on number of retransmit shreds ( #15694 )
2021-03-08 17:54:53 -08:00
sakridge
d09112fa6d
PoH batch size calibration ( #15717 )
2021-03-05 16:01:21 -08:00
Michael Vines
4a3ab77baf
Remove unused id field
2021-03-05 19:07:59 +00:00
Michael Vines
66b781eec3
Add 'unknown' health check state
2021-03-05 17:46:50 +00:00
Tyera Eulberg
7e65289729
Convert blockstore TransactionStatus column family to protobufs ( #15733 )
...
* Prevent panic if TransactionStatus can't be deserialized
* Convert Blockstore TransactionStatus column to protobuf
* Add compatability test
2021-03-05 09:05:35 -07:00
Michael Vines
bd13262b42
Add validator startup process reporting before RPC is available
2021-03-05 08:03:36 -08:00
Michael Vines
24ab84936e
Break up RPC API into three categories: minimal, full and admin
2021-03-04 16:39:44 -08:00
Jeff Washington (jwash)
34bebb7d09
report execution details in replay time ( #15693 )
2021-03-04 11:38:12 -06:00
Jeff Washington (jwash)
be35c1c1b7
add execute detail timings ( #15638 )
2021-03-03 17:07:45 -06:00
behzad nouri
658951e680
sends only the latest vote of each validator to the banking stage ( #15629 )
2021-03-03 19:07:16 +00:00
carllin
aacb28c453
Only report metrics every second ( #15652 )
2021-03-03 10:58:47 -08:00
sakridge
830be855dc
Forward and hold packets ( #15634 )
2021-03-03 10:23:05 -08:00
Tyera Eulberg
19ac79b5cc
Deprecate UiTokenAmount::ui_amount ( #15616 )
...
* Add TokenAmount::ui_amount_string
* Fixup solana-tokens
* Update docs
2021-03-02 22:51:41 -07:00
Tyera Eulberg
a4f0033bd7
Remove ValidatorConfig derive Clone, and fix local-cluster tests ( #15647 )
...
* Remove ValidatorConfig derive Clone
* Add local-cluster ValidatorConfig helpers
* Fix benches
2021-03-03 04:21:30 +00:00
behzad nouri
0bd0084b0d
adds more metrics for tx counts and batch sizes ( #15642 )
2021-03-03 01:28:15 +00:00
behzad nouri
416ea38028
adds metrics for the size and number of batches in bank_send_loop ( #15627 )
2021-03-02 15:44:35 +00:00
Greg Fitzgerald
2463cc1e6a
Fix typos ( #15610 )
2021-03-02 06:36:49 -08:00
Michael Vines
640e36287e
Move ValidatorExit into ValidatorConfig, making it accessible from the solana-validator crate
2021-03-01 16:49:56 -08:00
sakridge
f1223fb783
Lower blockstore processor error severity ( #15578 )
2021-03-01 14:57:37 -08:00
Trent Nelson
ef63ffce7d
Bump ed25519-dalek to 1.0.1
2021-03-01 21:10:46 +00:00
carllin
ae96ba3459
Plumb slot update pubsub notifications ( #15488 )
2021-02-28 23:29:11 -08:00
behzad nouri
f7a049f87f
coalesces vote packets into one Packets ( #15566 )
2021-02-26 23:23:08 +00:00
sakridge
05409e51ce
Increase tpu coalescing and add parameter ( #15536 )
...
Should create larger entries on average
2021-02-26 09:15:45 -08:00
behzad nouri
5a9896706c
indexes epoch slots in crds table ( #15459 )
...
ClusterInfo::get_epoch_slots_since scans the entire crds table to obtain
epoch-slots inserted since a timestamp:
https://github.com/solana-labs/solana/blob/013daa8f4/core/src/cluster_info.rs#L1245-L1262
The alternative is to index epoch-slots in crds table ordered by their
insert timestamp.
2021-02-26 14:12:04 +00:00
Tyera Eulberg
1ad2c9f741
Revert "Make UiTokenAmount::ui_amount a String ( #15447 )" ( #15542 )
...
This reverts commit d14374bc9f
.
2021-02-25 21:53:40 +00:00
Michael Vines
5b54aed1c0
Speed up getLeaderSchedule
2021-02-24 11:17:25 -08:00
Justin Starry
61ed980ac0
Fix received notifications for gossip signature subscriptions ( #15506 )
2021-02-24 16:59:22 +08:00
carllin
c2e8814dce
Add limit and shrink policy for recycler ( #15320 )
2021-02-24 00:15:58 -08:00
Tyera Eulberg
52f2d425e5
Count if optimistically confirmed slot is already rooted ( #15492 )
2021-02-23 22:03:22 +00:00
sakridge
1b59b163dd
Add max retransmit and shred insert slot ( #15475 )
2021-02-23 13:06:33 -08:00
Michael Vines
4b0114b991
Limit the number of getProgramAccounts filters
2021-02-23 18:43:22 +00:00
Tyera Eulberg
d14374bc9f
Make UiTokenAmount::ui_amount a String ( #15447 )
...
* Make UiTokenAmount::ui_amount a String
* Fixup solana-tokens
* Ignore spl downstream-project
2021-02-22 13:05:45 -07:00
Ryo Onodera
5ccaa6336a
Print original error from accounts dir remove ( #15458 )
2021-02-22 21:24:09 +09:00
Ivan Mironov
013daa8f47
RPC: Improve snapshot path sanitization
2021-02-20 13:06:07 -08:00
Michael Vines
5df36aec7d
Pacify clippy
2021-02-19 20:08:41 -08:00
Michael Vines
fd3b71a2c6
cargo fmt
2021-02-19 20:08:41 -08:00
behzad nouri
aa3aac766f
adds metrics for inbound/outbound gossip packets counts ( #15407 )
2021-02-19 22:49:35 +00:00
Justin Starry
15bbe6436d
Bump SPL token version to v3.1.0 ( #15429 )
...
* Bump SPL token version to v3.1.0
* Cargo.lock
2021-02-19 09:15:41 +00:00
Tyera Eulberg
170cb792eb
Return blockstore error if previous_blockhash cannot be determined ( #15382 )
...
* Return blockstore error if previous_blockhash cannot be determined
* Add require_previous_blockshash flag
2021-02-18 01:04:52 +00:00
Trent Nelson
7f7370c306
Re-allow clippy::integer_arithmetic at crate-level
2021-02-17 13:55:08 -07:00
Michael Vines
9ba69a7381
Adapt to fs_extra 1.2.0
2021-02-17 12:44:58 -08:00
sakridge
b24cb9840e
Speedup ledger cleanup test ( #15304 )
...
Just clone to produce shreds and use a separate insert thread.
2021-02-17 08:59:25 -08:00
Tyera Eulberg
98e3e570d2
Add --force arg for bigtable upload
2021-02-16 17:24:07 -08:00
Jeff Washington (jwash)
ba02452d75
add validator flag no-accounts-db-index-hashing ( #15350 )
...
* add validator flag no_accounts_db_index_hashing
* add validator flag no_accounts_db_index_hashing
2021-02-16 21:13:48 +00:00
behzad nouri
f79c9d4094
adds an upper bound on cluster-slots size ( #15300 )
...
https://github.com/solana-labs/solana/issues/14366#issuecomment-769096305
2021-02-16 21:12:13 +00:00
behzad nouri
076c20f1ca
checks that prune-messages have the same inner/outer pubkey ( #15352 )
2021-02-16 21:06:18 +00:00
Trent Nelson
b3256aa722
Bump rand_core to 0.6.2
...
https://rustsec.org/advisories/RUSTSEC-2021-0023
2021-02-15 22:51:35 +00:00
sakridge
5b8f046c67
More configurable rocksdb compaction ( #15213 )
...
rocksdb compaction can cause long stalls, so
make it more configurable to try and reduce those stalls
and also to coordinate between multiple nodes to not induce
stall at the same time.
2021-02-14 10:16:30 -08:00
carllin
629dcd0f39
Cleanup buffered packets ( #15210 )
2021-02-12 03:27:37 -08:00
carllin
990bb426a9
Fix flaky test test_concurrent_snapshot_packaging ( #15252 )
2021-02-11 16:03:51 -08:00
Josh
4013f91dbe
RPC: add caching to getLargestAccounts ( #15154 )
...
* introduce get largest accounts cache
* remove cache size and change hash key
* remove eq and hash derivation from commitment config
* add slot to the cache
2021-02-11 11:32:46 -08:00
Tyera Eulberg
948819dfa8
Use spl-token-mint secondary index for relevant getProgramAccounts requests ( #15219 )
2021-02-09 15:49:13 -07:00
Tyera Eulberg
da6753b8c0
Warp timestamp and extend max-allowable-drift for accommodate slow blocks ( #15204 )
...
* Remove timestamp_correction feature gating
* Remove timestamp_bounding feature gating
* Remove unused deprecated ledger code
* Remove unused deprecated unbounded-timestamp code
* Enable independent adjustment of fast/slow timestamp bounding
* Update timestamp bounds to 25% fast, 80% slow; warp timestamp
* Update bank hash test
* Add PR number to feature
Co-authored-by: Michael Vines <mvines@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-09 15:49:00 -07:00
behzad nouri
2758588ddd
uses btree-map instead of hash-map for cluster-slots ( #15194 )
...
retain traverses all values in the hashmap which is slow:
https://github.com/solana-labs/solana/blob/88f22c360/core/src/cluster_slots.rs#L45
btree-map instead allows more efficient prunning there.
In addition there is potential race condition here:
https://github.com/solana-labs/solana/blob/88f22c360/core/src/cluster_slots.rs#L68-L74
If another thread inserts a value at the same slot key between the read
and write lock, current thread will discard the inserted value.
2021-02-09 22:04:41 +00:00
Michael Vines
f063f02c41
solana-test-validator now uses the BPF JIT by default, `--no-bpf-jit` to disable
2021-02-09 20:24:14 +00:00
Michael Vines
2caed6bd4c
Complete --rpc-pubsub-max-in-buffer-capacity/--rpc-pubsub-max-out-buffer-capacity plumbing
2021-02-09 07:24:30 +00:00
behzad nouri
b6f231b60e
removes locked pubkey references ( #15152 )
2021-02-08 02:07:00 +00:00
sakridge
bbae23358c
ledger-tool cleanup and additions ( #15179 )
...
* Plumb allow-dead-slots to ledger-tool verify
* ledger-tool cleanup and add some useful missing args
Print root slots and how many unrooted past last root.
2021-02-06 17:26:42 -08:00
Jeff Washington (jwash)
fbf9dc47e9
Threadpool2 ( #15151 )
...
* rework thread pool for hash calculation
* rename
2021-02-05 18:48:16 -06:00
Jeff Washington (jwash)
fabecdc86c
use thread pool for non-index hash calculations ( #15149 )
2021-02-05 19:48:55 +00:00
behzad nouri
6fd5ec0e4c
caches descendants in bank forks ( #15107 )
2021-02-05 18:00:45 +00:00
Ryo Onodera
c69027e5aa
chore: bump serde from 1.0.118 to 1.0.122 ( #15126 )
...
* chore: bump serde from 1.0.118 to 1.0.122
* Update programs/bpf/Cargo.lock
2021-02-05 19:23:24 +09:00
Tyera Eulberg
d1563f0ccd
Bump tonic, prost, tarpc, tokio ( #15013 )
...
* Update tonic & prost, and regenerate proto
* Reignore doc code
* Revert pull #14367 , but pin tokio to v0.2 for jsonrpc
* Bump backoff and goauth -> and therefore tokio
* Bump tokio in faucet, net-utils
* Bump remaining tokio, plus tarpc
2021-02-05 00:21:53 -07:00
Jeff Washington (jwash)
600ff0d915
calculate hash from store instead of index ( #15034 )
...
* calculate hash from store instead of index
* restore update hash in abs
2021-02-04 09:00:33 -06:00
behzad nouri
86467d825a
removes pubkey references ( #15050 )
2021-02-03 23:02:11 +00:00
behzad nouri
0ad063f4e9
adds flag to disable duplicate instance check ( #15006 )
2021-02-03 16:26:17 +00:00
Tyera Eulberg
98aa1fa4ea
Upgrade jsonrpc crates to v17.0.0 ( #15018 )
...
* Upgrade to jsonrpc 17.0.0
* Fix test
* tree
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-02 19:53:08 -07:00
dependabot[bot]
1df93fa2be
chore: bump serde from 1.0.112 to 1.0.118 ( #14828 )
...
* chore: bump serde from 1.0.112 to 1.0.122
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.112 to 1.0.122.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.122 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Update frozen_abi digest following serde update
* Revert "chore: bump serde from 1.0.112 to 1.0.122"
This reverts commit a3ef4442a4c985144ae2bd7ceaf8899a7ab8d7c0.
* Revert "[auto-commit] Update all Cargo lock files"
This reverts commit c41c3b005fb1ccade55155302c52cd5736c4b55f.
* chore: bump serde from 1.0.112 to 1.0.118
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.112 to 1.0.118.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.118 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Remove serum-dex pinning
* blind commit!
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-02-02 23:28:16 +09:00
Tyera Eulberg
cbb8b79a60
Add validator flag to opt in to cpi and logs storage ( #14922 )
...
* Add validator flag to opt in to cpi and logs storage
* Default TestValidator to opt-in; allow using in multinode-demo
* No clone
Co-authored-by: Carl Lin <carl@solana.com>
2021-02-01 14:00:51 -07:00
Tom Parker-Shemilt
01230a0105
Remove serial_test_derive dependency ( #14891 )
2021-01-28 22:35:31 -07:00
Ryo Onodera
d6873b82ab
Remove potentially too costly Packets::default() ( #14821 )
...
* Remove potentially too costly Packets::default()
* Fix test...
* Restore Packets::default()
* Restore Packets::default() more
2021-01-29 09:32:38 +09:00
Michael Vines
119e2c75dd
cli now supports a custodian for stake authorize operations
2021-01-26 11:48:28 -08:00
Tyera Eulberg
ffa5c7dcc8
Deprecate commitment variants ( #14797 )
...
* Deprecate commitment variants
* Add new CommitmentConfig builders
* Add helpers to avoid allowing deprecated variants
* Remove deprecated transaction-status code
* Include new commitment variants in runtime commitment; allow deprecated as long as old variants persist
* Remove deprecated banks code
* Remove deprecated variants in core; allow deprecated in rpc/rpc-subscriptions for now
* Heavier hand with rpc/rpc-subscription commitment
* Remove deprecated variants from local-cluster
* Remove deprecated variants from various tools
* Remove deprecated variants from validator
* Update docs
* Remove deprecated client code
* Add new variants to cli; remove deprecated variants as possible
* Don't send new commitment variants to old clusters
* Retain deprecated method in test_validator_saves_tower
* Fix clippy matches! suggestion for BPF solana-sdk legacy compile test
* Refactor node version check to handle commitment variants and transaction encoding
* Hide deprecated variants from cli help
* Add cli App comments
2021-01-26 19:23:07 +00:00
behzad nouri
d1df9da7d3
fixes test_filter_current flakiness ( #14816 )
2021-01-25 15:57:46 +00:00
Ryo Onodera
bd0433c373
Clean up VerifiedVotePackets ( #14822 )
2021-01-25 09:01:47 +00:00
behzad nouri
e1021d9f83
removes redundant epoch stakes cache in retransmit ( #14781 )
...
Following d6d76219b
, staked nodes computed from vote accounts are
already cached in runtime::Stakes, so the caching in retransmit_stage is
redundant.
2021-01-24 21:15:09 +00:00
behzad nouri
491b059755
broadcasts duplicate shreds through gossip ( #14699 )
2021-01-24 15:47:43 +00:00
Michael Vines
cbffab7850
Upgrade to Rust v1.49.0
2021-01-23 19:16:36 -08:00
Michael Vines
bf1943e489
Add solana-test-validator --warp-slot argument
2021-01-22 21:17:02 -08:00
Michael Vines
cbb9ac19b9
Add ability to clone accounts from an RPC endpoint
2021-01-22 13:29:36 -08:00
Tyera Eulberg
71e9958e06
Rpc: Add custom error for BigTable data not found ( #14762 )
...
* Expose not-found bigtable error
* Add custom rpc error for bigtable data not found
* Return custom rpc error when bigtable block is not found
* Generalize long-term storage
2021-01-22 04:40:47 +00:00
behzad nouri
e4da6761a7
fixes test_filter_current flakiness ( #14749 )
2021-01-21 21:53:10 +00:00
behzad nouri
8e581601d6
patches crds vote-index assignment bug ( #14438 )
...
If tower is full, old votes are evicted from the front of the deque:
https://github.com/solana-labs/solana/blob/2074e407c/programs/vote/src/vote_state/mod.rs#L367-L373
whereas recent votes if expire are evicted from the back:
https://github.com/solana-labs/solana/blob/2074e407c/programs/vote/src/vote_state/mod.rs#L529-L537
As a result, from a single tower_index scalar, we cannot infer which crds-vote
should be overwritten:
https://github.com/solana-labs/solana/blob/2074e407c/core/src/crds_value.rs#L576
In addition there is an off by one bug in the existing code. tower_index is
bounded by MAX_LOCKOUT_HISTORY - 1:
https://github.com/solana-labs/solana/blob/2074e407c/core/src/consensus.rs#L382
So, it is at most 30, whereas MAX_VOTES is 32:
https://github.com/solana-labs/solana/blob/2074e407c/core/src/crds_value.rs#L29
Which means that this branch is never taken:
https://github.com/solana-labs/solana/blob/2074e407c/core/src/crds_value.rs#L590-L593
so crds table alwasys keeps 29 **oldest** votes by wallclock, and then
only overrides the 30st one each time. (i.e a tally of only two most
recent votes).
2021-01-21 13:08:07 +00:00
Michael Vines
04ce33a04e
Ensure sanitary transactions
2021-01-20 23:59:32 -08:00
Ryo Onodera
dcaa025822
Configure Bigtable's timeout, enabling by default ( #14657 )
...
* Configure bigtable's timeout when read-only
* Review comments
* Apply nits (thanks!)
Co-authored-by: Michael Vines <mvines@gmail.com>
* Timeout in the streamed decoding as well
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-01-19 13:57:16 +00:00
behzad nouri
b5fd0ed859
rewrites turbine retransmit peers computation ( #14584 )
2021-01-19 04:18:47 +00:00
behzad nouri
c6ae0667e6
feature gates turbine retransmit peers patch ( #14631 )
2021-01-19 04:16:19 +00:00
Michael Vines
5d9dc609b1
Rename RpcNodeUnhealthy error to NodeUnhealthy, generalize `getHealth` RPC error object for the future
2021-01-18 19:57:25 -08:00
Tyera Eulberg
cbf8ef7480
Make Bigtable::get_confirmed_blocks inclusive of requested start_slot and end_slot ( #14651 )
...
* Fix off-by-one error
* Filter out blocks greater than end slot
2021-01-18 19:14:10 -07:00
carllin
5f14f45282
More generic accounts purge functions ( #14595 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-17 20:31:03 -08:00
Ryo Onodera
8d4ab1bab1
Clean up and add comment
2021-01-17 19:59:59 -08:00
Ryo Onodera
5cf9094bb9
WIP fix the occasional stuck RPC request
2021-01-17 19:59:59 -08:00
Michael Vines
4003f86f04
Add `getSnapshotSlot` RPC method
2021-01-16 19:31:21 +00:00
Michael Vines
dacb95083d
Add `getHealth` RPC method
2021-01-16 10:10:48 -08:00
Tyera Eulberg
9a89689ad3
Use optimistic confirmation in getSignatureStatuses, and various downstream client methods ( #14430 )
...
* Add optimistically_confirmed field to TransactionStatus
* Update docs
* Convert new field to confirmation_status
* Update docs to confirmationStatus
* Update variants
* Update docs
* Just Confirmed
2021-01-15 16:05:05 +00:00
dependabot[bot]
4ecc0f4fa6
chore: bump rayon from 1.4.1 to 1.5.0 ( #14177 )
...
* chore: bump rayon from 1.4.1 to 1.5.0
Bumps [rayon](https://github.com/rayon-rs/rayon ) from 1.4.1 to 1.5.0.
- [Release notes](https://github.com/rayon-rs/rayon/releases )
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md )
- [Commits](https://github.com/rayon-rs/rayon/compare/rayon-core-v1.4.1...rayon-core-v1.5.0 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-01-15 18:07:35 +09:00
Michael Vines
1c2ae15b1d
Improve solana-test-validator output
2021-01-14 23:07:24 -08:00
sakridge
907f518f6d
Add load/execute/store timings ( #14561 )
2021-01-14 14:14:16 -08:00
behzad nouri
cfcca1cd3c
patches bug in turbine's neighbors computation ( #14565 )
...
Removing local node's index early from the set here:
https://github.com/solana-labs/solana/blob/e1b59ded4/core/src/retransmit_stage.rs#L346
distorts the order of nodes depending on which node is computing the
turbine fan-out tree, and results in incorrect neighbors computation.
2021-01-13 22:25:29 +00:00
Michael Vines
11daaadc93
Add --rpc-threads argument
2021-01-13 13:34:46 -08:00
Tyera Eulberg
e1b59ded4b
Use leader_forward_count for tx retries too ( #14547 )
2021-01-13 11:14:22 -07:00
Jeff Washington (jwash)
935dfdf0f6
fill in timing gaps in replay_stage ( #14550 )
...
* fill in timing gaps in replay_stage
* add replay_stage bank_count metric
* formatting
* handle another gap
* cleanup wait_receive_time to be more straightforward
2021-01-13 10:08:53 -06:00
carllin
6dfad0652f
Cache account stores, flush from AccountsBackgroundService ( #13140 )
2021-01-11 17:00:23 -08:00
Tyera Eulberg
b0e6e29527
Update timestamp max allowable drift to 50% of PoH ( #14531 )
...
* Repurpose warp-timestamp feature for general bump
* Change max_allowable_drift to 50%
* Fill in PR#
* Fix rpc test setup
2021-01-11 23:27:30 +00:00
Michael Vines
a95675a7ce
Avoid tmp snapshot backlog in SnapshotPackagerService under high load ( #14516 )
2021-01-11 10:21:15 -08:00
Michael Vines
7be6770808
Rename CompressionType to ArchiveFormat
2021-01-09 09:07:49 -08:00
Michael Vines
141e6706e6
Rename AccountsPackage::root to AccountsPackage::slot
2021-01-09 09:07:49 -08:00
behzad nouri
766195dded
limits number of crds values associated with a pubkey ( #14467 )
2021-01-08 18:54:40 +00:00
Jeff Washington (jwash)
773b21b34e
consolidate DEFAULT_HASHES_PER_TICK ( #14463 )
2021-01-07 09:49:42 -06:00
sakridge
c282586753
Add fixed tick rate adjustment ( #14447 )
2021-01-05 19:25:44 -08:00
Tyera Eulberg
ce1766d798
Enable program-id account index for supply calculations ( #14444 )
...
* Enable program-id account index for supply calculations
* Fixup comments
2021-01-06 02:42:33 +00:00
carllin
5affd8aa72
Add secondary indexes ( #14212 )
2020-12-31 18:06:03 -08:00
Michael Vines
4a3d217839
Only initialize BigTable upload service when requested
2020-12-31 17:40:51 -08:00
Michael Vines
0b23abd479
Bind RPC and faucet to 0.0.0.0
2020-12-31 07:55:26 +00:00
Michael Vines
3d077fb656
Revert "Upgrade in-tree tokio 0.2 usage to tokio 0.3"
...
This reverts commit 444ed768dc
.
2020-12-31 06:26:53 +00:00
Michael Vines
fb6c660cfd
Port ip-echo-server to tokio 0.3
2020-12-30 09:29:16 -08:00
Trent Nelson
fe667db910
validator: Add experimental flag to select PoH pinned core
2020-12-29 19:15:44 -07:00
Trent Nelson
82f61c0c4a
core: Update stale error message
2020-12-29 19:15:44 -07:00
sakridge
f8a4afc7c1
Fix flaky broadcast test ( #14329 )
2020-12-29 12:35:04 -08:00
sakridge
2074e407cd
Add poh speed check and tick speed calibration ( #14292 )
2020-12-29 09:35:57 -08:00
Michael Vines
444ed768dc
Upgrade in-tree tokio 0.2 usage to tokio 0.3
2020-12-29 09:23:23 -08:00
Tyera Eulberg
3f10fb993b
Retry durable-nonce transactions ( #14308 )
...
* Retry durable-nonce transactions
* Add metric to track durable-nonce txs in queue
* Populate send-tx-service initial addresses with tpu_address if empty (primarily for testing)
* Reinstate last_valid_slot check for durable-nonce txs; use arbitrary future slot
2020-12-29 09:48:43 -07:00
Michael Vines
9ddd6f08e8
Persist gossip contact info
2020-12-27 20:46:54 -08:00
sakridge
c693ffaa08
Fix subtraction overflow in metrics ( #14290 )
2020-12-27 16:26:22 -08:00
behzad nouri
2fd38d9912
indexes votes in crds table ( #14272 )
2020-12-27 13:31:05 +00:00
behzad nouri
49019c6613
obtains staked-nodes from the root-bank ( #14257 )
...
... as opposed to the working bank
2020-12-27 13:28:05 +00:00
Trent Nelson
5b903318b2
vote: Add helper for creating current-versioned states
2020-12-22 19:37:26 -07:00
Michael Vines
ace360ade2
Multiple entrypoint support
2020-12-22 18:35:31 -08:00
Michael Vines
3373082ffa
Update entrypoint contact info even when shred version adoption is not requested
2020-12-22 18:35:31 -08:00
behzad nouri
a14cfd660a
removes &Arc<Self> receivers ( #14234 )
2020-12-22 23:51:53 +00:00
sakridge
baa9602411
Add shrink paths ( #14238 )
2020-12-21 21:33:37 -08:00
behzad nouri
7b08cb1f0d
improves performance in replay-stage ( #14217 )
...
bank::vote_accounts returns a hash-map which is slow to iterate, but all uses
only require an iterator:
https://github.com/solana-labs/solana/blob/b3dc98856/runtime/src/bank.rs#L4300-L4306
Similarly, calculate_stake_weighted_timestamp takes a hash-map whereas it only
requires an iterator:
https://github.com/solana-labs/solana/blob/b3dc98856/sdk/src/stake_weighted_timestamp.rs#L21-L28
2020-12-21 19:18:19 +00:00
Michael Vines
57b03c5bc1
Restore Content-Length header for streaming snapshot download
2020-12-20 23:22:26 -08:00
Michael Vines
b3dc988564
Stream RPC snapshot downloads
2020-12-19 15:54:23 -08:00
Michael Vines
0090106f60
getBlockTime RPC method now falls back to BigTable in all cases
2020-12-18 20:51:49 +00:00
behzad nouri
691031fefd
limits number of crds values returned when responding to pull requests ( #13739 )
...
Crds values buffered when responding to pull-requests can be very large taking a lot of memory.
Added a limit for number of buffered crds values based on outbound data budget.
2020-12-18 18:45:12 +00:00
behzad nouri
6a3797e164
adds crds-value for broadcasting duplicate shreds through gossip ( #14133 )
...
In gossip, the header overhead we get from:
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/cluster_info.rs#L434-L435
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/crds_value.rs#L31-L36
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/crds_value.rs#L73
already exceeds SIZE_OF_NONCE in shreds. We also need aditional
meta-data (wallclock, source pubkey, ...). Which means that given the
SHRED_PAYLOAD_SIZE, we cannot fit all these in PACKET_DATA_SIZE:
https://github.com/solana-labs/solana/blob/de9ac43eb/ledger/src/shred.rs#L80
On top of that, we need 2 shred payloads as the proof of duplicate. So
each DuplicateShred crds value includes only a chunk of the payload,
along with the meta-data to reconstruct the full payload from the chunks
on the receiving end.
2020-12-18 14:32:43 +00:00
sakridge
a5db6399ad
Fix test_max_hashes ( #14189 )
2020-12-17 15:12:18 -08:00
sakridge
da7d1e2302
Improved Transaction Forwarding ( #13944 )
...
* Forwarding
* Dedupe leaders
* Use consistent commitment for last_valid_slot in rpc send_transaction
* Plumb rpc send_transaction options into solana-validator
* Extend num slots banking-stage holds forwarded txs
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-12-17 15:37:22 -07:00
behzad nouri
d6d76219b6
caches staked nodes computed from vote-accounts ( #13929 )
2020-12-17 21:22:50 +00:00
Michael Vines
0b92720fdb
Don't require increased open file limit in solana-test-validator
...
Travis CI in particular does not allow the open file limit to be
increased.
2020-12-16 20:43:08 -08:00
Trent Nelson
6875113d82
Bump SPL crates
2020-12-17 01:54:22 +00:00
Tyera Eulberg
ac0d32bc7e
Add blockstore skipped api ( #14145 )
...
* Add blockstore api to determine if a slot was skipped
* Return custom rpc error if slot is skipped
2020-12-16 13:40:36 -07:00
sakridge
d4a174fb7c
Partial shred deserialize cleanup and shred type differentiation ( #14094 )
...
* Partial shred deserialize cleanup and shred type differentiation in retransmit
* consolidate packet hashing logic
2020-12-15 16:50:40 -08:00
carllin
75e9e321de
Fix race between setting tick height and calculating accounts hash ( #14101 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-12-15 12:45:40 -08:00
Michael Vines
1e977ac025
Bump version to v1.6.0
2020-12-15 18:28:04 +00:00
behzad nouri
c2b7115031
indexes crds values associated with a pubkey ( #14088 )
...
record_labels returns all the possible labels for a record identified by
a pubkey, used in updating timestamp of crds values:
https://github.com/solana-labs/solana/blob/1792100e2/core/src/crds_value.rs#L560-L577
https://github.com/solana-labs/solana/blob/1792100e2/core/src/crds.rs#L240-L251
The code relies on CrdsValueLabel to be limited to a small deterministic
set of possible values for a fixed pubkey. As we expand crds values to
include duplicate shreds, this limits what the duplicate proofs can be
keyed by in the table.
In addition the computation of these labels is inefficient and will
become more so as duplicate shreds and more types of crds values are
added. An alternative is to maintain an index of all crds values
associated with a pubkey.
2020-12-15 01:49:22 +00:00
sakridge
5294f70189
Remove some non-warnings ( #14115 )
2020-12-14 16:26:17 -08:00
Michael Vines
7143aaa89b
Clippy
2020-12-14 08:03:29 -08:00
Michael Vines
cdd3e7d856
Remove solana-vote-signer ( #14099 )
2020-12-13 19:12:20 -08:00
Michael Vines
9f2d154588
Rework TestValidator API to be more like ProgramTest
2020-12-13 07:45:55 +00:00
dependabot[bot]
88d950e5a8
chore: bump log from 0.4.8 to 0.4.11 ( #13691 )
...
* chore: bump log from 0.4.8 to 0.4.11
Bumps [log](https://github.com/rust-lang/log ) from 0.4.8 to 0.4.11.
- [Release notes](https://github.com/rust-lang/log/releases )
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/log/compare/0.4.8...0.4.11 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-12-13 13:48:23 +09:00
carllin
55fc963595
Move slot cleanup to AccountsBackgroundService ( #13911 )
...
* Move bank drop to AccountsBackgroundService
* Send to ABS on drop instead, protects against other places banks are dropped
* Fix Abi
* test
Co-authored-by: Carl Lin <carl@solana.com>
2020-12-13 01:22:34 +00:00
Tyera Eulberg
8541ffa328
Add subscription cap ( #14011 )
...
* Add subscription cap
* Elide unwraps
2020-12-12 01:57:40 +00:00
Trent Nelson
546c6a4531
Override yanked miow crate to 0.2.2
2020-12-11 23:44:03 +00:00
Trent Nelson
277b6790c7
Override yanked net2 0.2.37
2020-12-11 23:44:03 +00:00
Ryo Onodera
09bd412b13
Wrong rpc inflation rate ( #14063 )
...
* Fix wrong inflation calculation in rpc
* Reorder
* Fix test
2020-12-11 07:42:39 +00:00
Michael Vines
4fba7e6865
Move validator_block_0_fee_workaround into TestValidator
2020-12-11 04:17:38 +00:00
Michael Vines
bbad3fe501
TestValidator now implements Drop, no need to close() it
2020-12-11 04:17:38 +00:00
Michael Vines
0a9ff1dc9d
Initial solana-test-validator command-line program
2020-12-11 04:17:38 +00:00
Josh
13db3eca9f
SPL token balance in transaction metadata ( #13673 )
...
* feat: store pre / post token balances
* move helper functions into separate include
* move token balance functionality to transaction-status crate
* fix blockstore processor test
* fix bigtable legacy test
* add caching to decimals
2020-12-10 19:25:07 -08:00
sakridge
aa2751e614
Check shred type in is_duplicate ( #14050 )
2020-12-10 18:20:08 -08:00
behzad nouri
409fe3bca1
adds the instance token to crds-labels for node-instance crds-values ( #14037 )
...
If a node "a" receives instance-info from node "b1" it will override any
instance-info associated with "b1" pubkey in its crds table. This makes
it less likely that when "b1" receives crds values from "a" (either
through pull or push), it sees other instances of itself (because node
"a" discarded them when it received "b1" instance info).
In order for the crds table to contain all instance-info associated with
the same pubkey at the same time, we need to add the instance tokens to
the keys in the crds table (i.e. the CrdsValueLabel).
2020-12-10 17:01:55 +00:00
sakridge
5c95d8e963
Shred filter ( #14030 )
2020-12-10 07:54:15 -08:00
sakridge
c5fe076432
Better dupe detection ( #13992 )
2020-12-09 23:14:31 -08:00
behzad nouri
1d267eae6b
std::process::exit to kill all threads
2020-12-09 10:24:23 -08:00
behzad nouri
895d7d6a65
removes RwLock on ClusterInfo.instance
2020-12-09 10:24:23 -08:00
behzad nouri
542198180a
pushes node-instance along with version early in gossip
2020-12-09 10:24:23 -08:00
behzad nouri
8cd5eb9863
checks for duplicate validator instances using gossip
2020-12-09 10:24:23 -08:00
sakridge
f6600810d7
Use LRU cache and blake3 hash of shreds to filter duplicates ( #13976 )
2020-12-07 16:42:39 -08:00
Michael Vines
6e9dbb4f6e
Add --rpc-max-multiple-accounts to override the getMultipleAccounts JSON RPC maximum
2020-12-07 16:31:01 -08:00
carllin
239a191612
Remove unneeded BankWeight fork choice ( #13978 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-12-07 13:47:14 -08:00
Tyera Eulberg
6ae4d2e5cb
Fix logsSubscribe ( #13996 )
2020-12-07 19:00:52 +00:00
Ryo Onodera
3d9d7557c8
core/validator: Wrap std::process:exit(1) for easier testing ( #13990 )
2020-12-07 16:43:03 +00:00
Alexander Meißner
a706706572
Validator CLI option to enable just-in-time compilation of BPF ( #13789 )
...
* Adds a CLI option to the validator to enable just-in-time compilation of BPF.
* Refactoring to use bpf_loader_program instead of feature_set to pass JIT flag from the validator CLI to the executor.
2020-12-07 09:49:55 +01:00
behzad nouri
6706f2b3bb
removes recursive read-locks on gossip ( #13973 )
...
ClusterInfo::tvu_peers acquires a read-lock on gossip:
https://github.com/solana-labs/solana/blob/f0e934145/core/src/cluster_info.rs#L1171-L1185
and so, ClusterInfo::repair_peers is recursively locking gossip for
read twice:
https://github.com/solana-labs/solana/blob/f0e934145/core/src/cluster_info.rs#L1202-L1223
But std::sync::RwLock is not re-entrant (recursive).
2020-12-06 15:14:49 +00:00
Tyera Eulberg
ca35bb3ac8
Report highest_confirmed_root and _slot in commitment metric ( #13964 )
2020-12-05 00:50:00 +00:00
carllin
34b68288c8
Fix propagation skip check ( #13933 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-12-03 12:31:38 -08:00
behzad nouri
c3048b451d
samples repair peers using WeightedIndex ( #13919 )
...
To output one random sample, weighted_best generates n random numbers:
https://github.com/solana-labs/solana/blob/f751a5d4e/core/src/weighted_shuffle.rs#L38-L63
WeightedIndex does so with only one random number:
https://github.com/rust-random/rand/blob/eb02f0e46/src/distributions/weighted_index.rs#L223-L240
Additionally, if the index is already constructed, it only does a total
of O(log(n)) amount of work; which can be achieved if RepairCache,
caches the weighted index:
https://github.com/solana-labs/solana/blob/f751a5d4e/core/src/serve_repair.rs#L83
Also, the repair-peers code can be reorganized to have fewer redundant
unlock-then-lock code.
2020-12-03 14:26:07 +00:00
Trent Nelson
404fc1570d
runtime: Replace `HashAgeKind` with `NonceRollbackInfo`
2020-12-02 20:10:08 +00:00
Tyera Eulberg
10c81a2448
Remove rpc_banks from validator ( #13882 )
...
* Remove rpc_banks from validator
* Bump abi-digest
2020-12-02 03:25:09 +00:00
Michael Vines
0a8bc347a1
Restore discover_cluster to avoid test panics
2020-12-01 17:58:28 -08:00
Michael Vines
3eece38ffa
Add expects() to improve error logs on join failures
2020-12-01 17:58:28 -08:00
Michael Vines
73111b005f
Reduce the number of snapshots
2020-12-01 11:13:37 -08:00
Tyera Eulberg
8fd1e55805
Add logging in check_blockstore_max_root ( #13887 )
2020-12-01 07:44:18 +00:00
Michael Vines
90d557d916
Strengthen EpochSlots sanitization
2020-11-30 14:40:25 -08:00
behzad nouri
e1793e5a13
caches vote-state de-serialized from vote accounts ( #13795 )
...
Gossip and other places repeatedly de-serialize vote-state stored in
vote accounts. Ideally the first de-serialization should cache the
result.
This commit adds new VoteAccount type which lazily de-serializes
VoteState from Account data and caches the result internally.
Serialize and Deserialize traits are manually implemented to match
existing code. So, despite changes to frozen_abi, this commit should be
backward compatible.
2020-11-30 17:18:33 +00:00
Michael Vines
43b82b31e5
More TestValidator cleanup
2020-11-26 08:56:25 +00:00
Michael Vines
b5f7e39be8
TestValidator public interface cleanup
2020-11-25 17:04:37 -08:00
Tyera Eulberg
4ff0f0949a
Separate blockstore checks for not (yet) rooted and cleaned up ( #13814 )
2020-11-25 22:59:38 +00:00
Michael Vines
4ef2da0ff0
Add `solana logs` command
2020-11-25 11:44:41 -08:00
sakridge
b70abdc645
Nonce updates ( #13799 )
...
* runtime: Add `FeeCalculator` resolution method to `HashAgeKind`
* runtime: Plumb fee-collected accounts for failed nonce tx rollback
* runtime: Use fee-collected nonce/fee account for nonced TX error rollback
* runtime: Add test for failed nonced TX accounts rollback
* Fee payer test
* fixup: replace nonce account when it pays the fee
* fixup: nonce fee-payer collect test
* fixup: fixup: clippy/fmt for replace...
* runtime: Test for `HashAgeKind::fee_calculator()`
* Clippy
Co-authored-by: Trent Nelson <trent@solana.com>
2020-11-24 23:53:51 -08:00
Michael Vines
215ddecaa5
Add base64+zstd encoding for RPC account data
2020-11-25 02:03:23 +00:00
Michael Vines
61ab2072bd
Clean up default commitment handling for subscriptions
2020-11-23 22:54:47 -08:00
Tyera Eulberg
7befad2f6d
Check SlotNotRooted if confirmed block not found in blockstore or bigtable ( #13776 )
2020-11-24 03:36:20 +00:00
behzad nouri
26bf2b7e45
processes pull-request callers only once per unique caller ( #13750 )
...
process_pull_requests acquires a write lock on crds table to update
records timestamp for each of the pull-request callers:
https://github.com/solana-labs/solana/blob/3087c9049/core/src/crds_gossip_pull.rs#L287-L300
However, pull-requests overlap a lot in callers and this function ends
up doing a lot of redundant duplicate work.
This commit obtains unique callers before acquiring an exclusive lock on
crds table.
2020-11-22 17:51:14 +00:00
sakridge
c1eb350c47
Allow contact debug interval to be adjusted ( #13737 )
2020-11-20 14:47:37 -08:00
Ryo Onodera
b74d7b5758
Fix fragile tests in prep of stake rewrite pr ( #13654 )
...
* Fix fragile tests in prep of stake rewrite pr
* Restore BOOTSTRAP_VALIDATOR_LAMPORTS where appropriate
* Further clean up
* Further clean up
* Aligh with other call site change
* Remove false warn!
* fix ci!
2020-11-20 17:21:03 +09:00
behzad nouri
a8c29505f0
sanitizes bloom filters to avoid division by zero ( #13714 )
...
Pull requests received over the wire can cause a validator to panic
because of division by zero in bloom filters:
https://github.com/solana-labs/solana/blob/af08ba93e/runtime/src/bloom.rs#L86-L88
2020-11-19 23:35:22 +00:00
dependabot[bot]
856693ac1f
chore: bump lru from 0.6.0 to 0.6.1
...
Bumps [lru](https://github.com/jeromefroe/lru-rs ) from 0.6.0 to 0.6.1.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases )
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/jeromefroe/lru-rs/compare/0.6.0...0.6.1 )
Signed-off-by: dependabot[bot] <support@github.com>
2020-11-19 14:28:50 -08:00
behzad nouri
b58f69297f
makes crds fields private ( #13703 )
...
Crds fields should maintain several invariants between themselves, so
exposing them as public fields can be bug prone. In addition these
invariants are asserted on every write:
https://github.com/solana-labs/solana/blob/9668dd85d/core/src/crds.rs#L138-L154
https://github.com/solana-labs/solana/blob/9668dd85d/core/src/crds.rs#L239-L262
which adds extra instructions and is not optimal. Should these fields be
private the asserts will be redundant.
2020-11-19 20:57:40 +00:00
behzad nouri
1ffab5de77
breaks prunes data into chunks to fit into packets ( #13613 )
...
Validator logs show that prune messages are dropped because they exceed
packet data size:
https://github.com/solana-labs/solana/blob/f25c969ad/perf/src/packet.rs#L90-L92
This can exacerbate gossip traffic by redundantly increasing push
messages across network. The workaround is to break prunes into smaller
chunks and send over in multiple messages.
2020-11-19 16:38:01 +00:00
Trent Nelson
f2a1a0ac5c
RPC: Demote missing block error to warning
...
It frightens the tourists
2020-11-19 04:54:49 +00:00
Tyera Eulberg
3e4acba72f
Quiet notification logs when no subscriptions ( #13629 )
2020-11-17 06:37:38 +00:00
Tyera Eulberg
ef99689592
Improve TestValidator instantiation ( #13627 )
...
* Add TestValidator::new_with_fees constructor, and warning for low bootstrap_validator_lamports
* Add logging to solana-tokens integration test to help catch low bootstrap_validator_lamports in the future
* Reasonable TestValidator mint_lamports
2020-11-16 23:27:36 -07:00
behzad nouri
5e8490ab9d
packs more crds-values in a single gossip packet ( #13500 )
...
split_gossip_messages:
https://github.com/solana-labs/solana/blob/a97c04b40/core/src/cluster_info.rs#L1536-L1574
splits crds-values into chunks to fit into a gossip packet. However it is
using a global upper-bound for the header-size across all protocols:
https://github.com/solana-labs/solana/blob/a97c04b40/core/src/cluster_info.rs#L90-L93
This can be wasteful as the specific gossip protocol can have smaller
header than this upper-bound (e.g. Protocol::PushMessage is 170 bytes
smaller). Adding more crds-values in one gossip packet can avoid the
overheads of separate packets and reduce total number of bytes sent over
the wire.
This commit updates the splitting function to take a max-chunk-size
argument. At call-site, this value is set to the size of the protocol
which the values are sent over.
2020-11-15 18:23:59 +00:00
behzad nouri
cbea9ebc34
indexes nodes' contact infos in crds table ( #13553 )
...
In several places in gossip code, the entire crds table is scanned only
to filter out nodes' contact infos. Currently on mainnet, crds table is
of size ~70k, while there are only ~470 nodes. So the full table scan is
inefficient. Instead we may maintain an index of only nodes' contact
infos.
2020-11-15 16:38:04 +00:00
Michael Vines
5d72e52ad0
Disable the PubSub vote subscription by default
...
The --rpc-pubsub-enable-vote-subscription flag may be used to enable it.
The current vote subscription is problematic because it emits a
notification for *every* vote, so hundreds a second in a real cluster.
Critically it's also missing information about *who* is voting,
rendering all those notifications practically useless.
Until these two issues can be resolved, the vote subscription is not
much more than a potential DoS vector.
2020-11-14 12:36:37 -08:00
Tyera Eulberg
88ae321d3f
Add counter metrics to rpc-subscriptions ( #13596 )
2020-11-14 12:40:24 -07:00
Michael Vines
baa6b3a261
Add stable program logging for BPF and native programs
2020-11-14 08:26:01 -08:00
Tyera Eulberg
34bf80ba9c
Send pubsub metrics to metrics server ( #13584 )
2020-11-13 19:31:23 +00:00
sakridge
c1f3f9d27b
Stop searching for incorrect shred version after a minute ( #13512 )
2020-11-12 14:01:13 -08:00
behzad nouri
4e4e12b384
filters out offline nodes from pull options ( #13533 )
...
Inactive nodes are still observing incoming gossip traffic:
https://discord.com/channels/428295358100013066/670512312339398668/776140351291260968
likely because of pull-requests.
Previous related issues and commits:
https://github.com/solana-labs/solana/issues/12409
https://github.com/solana-labs/solana/pull/12620
https://github.com/solana-labs/solana/pull/12674
This commit implements same logic as
https://github.com/solana-labs/solana/pull/12674
to exclude inactive nodes from pull options, with the same periodic
retry logic for offline staked nodes in order to mitigate eclipse
attack.
2020-11-12 16:09:37 +00:00
carllin
9821a7754c
Discard pre hard fork persisted tower if hard-forking ( #13536 )
...
* Discard pre hard fork persisted tower if hard-forking
* Relax config.require_tower
* Add cluster test
* nits
* Remove unnecessary check
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-12 23:29:04 +09:00
Ryo Onodera
89b474e192
Fix slow/stuck unstaking due to toggling in epoch ( #13501 )
...
* Fix slow/stuck unstaking due to toggling in epoch
* nits
* nits
* Add stake_program_v2 feature status check to cli
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-11-11 14:11:57 -07:00
Trent Nelson
38f15e41b5
Validator: Periodically log what we're waiting for during `--wait-for-supermajority`
2020-11-11 20:03:26 +00:00
sakridge
70c4626efe
Fix signature access ( #13491 )
2020-11-10 08:35:03 -08:00
Justin Starry
a97c04b400
Send RPC notification when account is deleted ( #13440 )
...
* Send RPC notification when account is deleted
* Remove unwrap
2020-11-10 19:48:42 +08:00
sakridge
b4cf968e14
Add back shredding broadcast stats ( #13463 )
2020-11-09 23:04:27 -08:00
sakridge
c644b05c54
Fix avx check with newest nightly compiler ( #13465 )
2020-11-09 08:04:34 -08:00
behzad nouri
73ac104df2
propagates errors out of Packet::from_data ( #13445 )
...
Packet::from_data is ignoring serialization errors:
https://github.com/solana-labs/solana/blob/d08c3232e/sdk/src/packet.rs#L42-L48
This is likely never useful as the packet will be sent over the wire
taking bandwidth but at the receiving end will either fail to
deserialize or it will be invalid.
This commit will propagate the errors out of the function to the
call-site, allowing the call-site to handle the error.
2020-11-08 15:10:03 +00:00
behzad nouri
7f4debdad5
drops older gossip packets when load shedding ( #13364 )
...
Gossip drops incoming packets when overloaded:
https://github.com/solana-labs/solana/blob/f6a73098a/core/src/cluster_info.rs#L2462-L2475
However newer packets are dropped in favor of the older ones.
This is probably not ideal as newer packets are more likely to contain
more recent data, so dropping them will keep the validator state
lagging.
2020-11-05 17:14:28 +00:00
behzad nouri
8f0796436a
shares the lock on gossip when processing prune messages ( #13339 )
...
Processing prune messages acquires an exclusive lock on gossip:
https://github.com/solana-labs/solana/blob/55b0428ff/core/src/cluster_info.rs#L1824-L1825
This can be reduced to a shared lock if active-sets are changed to use
atomic bloom filters:
https://github.com/solana-labs/solana/blob/55b0428ff/core/src/crds_gossip_push.rs#L50
2020-11-05 15:42:00 +00:00
behzad nouri
118ce47b97
measures processing time of each kind of gossip packets ( #13366 )
2020-11-05 15:34:34 +00:00
Michael Vines
8838a55a1a
Bump spl-token and spl-memo crate versions
2020-11-04 21:44:33 +00:00
behzad nouri
10fa4f45ab
uses thread-pool when handling push messages ( #13338 )
...
From runtime profiles, the majority time of solana-listen thread:
https://github.com/solana-labs/solana/blob/55b0428ff/core/src/cluster_info.rs#L2720
is spent handling push messages. The code here:
https://github.com/solana-labs/solana/blob/55b0428ff/core/src/cluster_info.rs#L2272-L2364
may utilize the idle gossip thread-pool.
2020-11-04 19:15:58 +00:00
sakridge
0d663158d0
Reduce repair_stats debug ( #13393 )
2020-11-04 10:32:48 -08:00
Tyera Eulberg
eb2560e782
Prevent block times from ever going backward
2020-10-31 21:30:42 -07:00
Tyera Eulberg
90778615f6
Use bounded timestamp-correction when feature enabled
2020-10-31 21:30:42 -07:00
sakridge
8415c22b59
Reduce debug ( #13319 )
2020-10-30 16:18:44 -07:00
Ryo Onodera
1df15d85c3
Fix tower/blockstore unsync due to external causes ( #12671 )
...
* Fix tower/blockstore unsync due to external causes
* Add and clean up long comments
* Clean up test
* Comment about warped_slot_history
* Run test_future_tower with master-only/master-slave
* Update comments about false leader condition
2020-10-30 19:31:23 +09:00
Michael Vines
df8dab9d2b
Native/builtin programs now receive an InvokeContext
2020-10-29 21:45:24 -07:00
Greg Fitzgerald
ca00197009
Upgrade tarpc and tokio ( #13293 )
2020-10-29 19:21:18 -06:00
behzad nouri
3738611f5c
adds more parallel processing to gossip packets handling ( #12988 )
2020-10-29 15:17:19 +00:00
behzad nouri
be80f6d5c5
excludes origin from prune set ( #13204 )
...
On the receiving end, prune messages are ignored if the origin points to
the node itself:
https://github.com/solana-labs/solana/blob/631f029fe/core/src/crds_gossip_push.rs#L285-L295
So to avoid sending these over the wire, the requester can exclude
origin from the prune set.
2020-10-29 12:50:58 +00:00
Jack May
c458d4b213
move Account to solana-sdk ( #13198 )
2020-10-28 22:01:07 -07:00
behzad nouri
ae91270961
implements ping-pong packets between nodes ( #12794 )
...
https://hackerone.com/reports/991106
> It’s possible to use UDP gossip protocol to amplify DDoS attacks. An attacker
> can spoof IP address in UDP packet when sending PullRequest to the node.
> There's no any validation if provided source IP address is not spoofed and
> the node can send much larger PullResponse to victim's IP. As I checked,
> PullRequest is about 290 bytes, while PullResponse is about 10 kB. It means
> that amplification is about 34x. This way an attacker can easily perform DDoS
> attack both on Solana node and third-party server.
>
> To prevent it, need for example to implement ping-pong mechanism similar as
> in Ethereum: Before accepting requests from remote client needs to validate
> his IP. Local node sends Ping packet to the remote node and it needs to reply
> with Pong packet that contains hash of matching Ping packet. Content of Ping
> packet is unpredictable. If hash from Pong packet matches, local node can
> remember IP where Ping packet was sent as correct and allow further
> communication.
>
> More info:
> https://github.com/ethereum/devp2p/blob/master/discv4.md#endpoint-proof
> https://github.com/ethereum/devp2p/blob/master/discv4.md#wire-protocol
The commit adds a PingCache, which maintains records of remote nodes
which have returned a valid response to a ping message, and on-the-fly
ping messages pending a pong response from the remote node.
When handling pull-requests, those from addresses which have not passed
the ping-pong check are filtered out, and additionally ping packets are
added for addresses which need to be (re)verified.
2020-10-28 17:03:02 +00:00
carllin
f96ab5a818
Fix log ( #13207 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-27 18:56:57 -07:00
Tyera Eulberg
39686ef098
Use bank timestamp to populate Blockstore::blocktime_cf when correction active ( #13158 )
2020-10-26 19:23:45 +00:00
behzad nouri
4bfda3e766
marks pull request creation time only once per peer ( #13113 )
...
mark_pull_request_creation time requires an exclusive lock on gossip:
https://github.com/solana-labs/solana/blob/16944e218/core/src/cluster_info.rs#L1547-L1548
Current code is redundantly marking each peer once for each request.
There are at most only 2 unique peers, whereas there are hundreds of
requests per each. So the lock is acquired hundreds of time longer than
necessary.
2020-10-26 17:11:31 +00:00
Ryo Onodera
66c7a98009
Allow existence of vote on root in saved tower ( #13135 )
2020-10-26 11:08:20 +09:00
Michael Vines
a4956844bd
Update frozen_abi hashes
...
The movement of files in sdk/ caused ABI hashes to change
2020-10-24 08:37:55 -07:00
Josh
766406fd23
add precompile verification to simulate_transaction ( #13080 )
2020-10-23 20:47:51 -07:00
Ryo Onodera
0264147d42
Clean up opt conf verifier and vote state tracker ( #13081 )
...
* Clean up opt conf verifier and vote state tracker
* Update test to follow new message and some knob
* Rename
2020-10-24 10:19:12 +09:00
behzad nouri
37c8842bcb
scans crds table in parallel for finding old labels ( #13073 )
...
From runtime profiles, the majority time of ClusterInfo::handle_purge
https://github.com/solana-labs/solana/blob/0776fa05c/core/src/cluster_info.rs#L1605-L1626
is spent scanning crds table finding old labels:
https://github.com/solana-labs/solana/blob/0776fa05c/core/src/crds.rs#L175-L197
This can be done in parallel given that gossip thread-pool:
https://github.com/solana-labs/solana/blob/0776fa05c/core/src/cluster_info.rs#L1637-L1641
is idle when handle_purge is invoked:
https://github.com/solana-labs/solana/blob/0776fa05c/core/src/cluster_info.rs#L1681
2020-10-23 14:17:37 +00:00
Justin Starry
c95f6c4b83
Remove spammy invalid rpc log ( #13100 )
2020-10-23 07:05:29 +00:00
Justin Starry
8b0242a5d8
Allow nodes to advertise a different rpc address over gossip ( #13053 )
...
* Allow nodes to advertise a different rpc address over gossip
* Feedback
2020-10-22 03:31:48 +00:00
Michael Vines
959880db60
Remove unused pubkey::Pubkey imports
2020-10-21 19:08:13 -07:00
Michael Vines
17c391121a
Run `codemod --extensions rs Hash::new_rand solana_sdk: #️⃣ :new_rand`
2020-10-21 19:08:13 -07:00
Michael Vines
7bc073defe
Run `codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand`
2020-10-21 19:08:13 -07:00
Ryo Onodera
0776fa05c7
Add ledger-tool dead-slots and improve purge a lot ( #13065 )
...
* Add ledger-tool dead-slots and improve purge a lot
* Reduce batch size...
* Add --dead-slots-only and fixed purge ordering
2020-10-21 17:45:21 +00:00
Ryo Onodera
efdb560e97
Various clean-ups before assert adjustment ( #13006 )
...
* Various clean-ups before assert adjustment
* oops
2020-10-21 10:26:20 +09:00
Michael Vines
6858950f76
Remove frozen ABI modules from solana-sdk
2020-10-20 16:11:30 -07:00
Ryo Onodera
c0675968b1
Support Debug Bank ( #13017 )
2020-10-21 01:05:45 +09:00
Trent Nelson
3b3f7341fa
validator: Activate RPC before halting on slot
2020-10-20 02:09:07 +00:00
behzad nouri
75d62ca095
improves threads' utilization in processing gossip packets ( #12962 )
...
ClusterInfo::process_packets handles incoming packets in a thread_pool:
https://github.com/solana-labs/solana/blob/87311cce7/core/src/cluster_info.rs#L2118-L2134
However, profiling runtime shows that threads are not well utilized and
a lot of the processing is done sequentially.
This commit redistributes the work done in parallel. Testing on a gce
cluster shows 20%+ improvement in processing gossip packets with much
smaller variations.
2020-10-19 19:03:38 +00:00
Ryo Onodera
54517ea454
Follow up to persistent tower with tests and API cleaning ( #12350 )
...
* Follow up to persistent tower
* Ignore for now...
* Hard-code validator identities for easy reasoning
* Add a test for opt. conf violation without tower
* Fix compile with rust < 1.47
* Remove unused method
* More move of assert tweak to the asser pr
* Add comments
* Clean up
* Clean the test addressing various review comments
* Clean up a bit
2020-10-19 16:37:03 +09:00
Ryo Onodera
fd8ec27fe8
Another some tower logging improvements ( #12940 )
2020-10-16 14:44:07 +09:00
behzad nouri
48283161c3
passes through feature-set to gossip requests handling ( #12878 )
...
* passes through feature-set to down to gossip requests handling
* takes the feature-set from root_bank instead of working_bank
2020-10-15 20:54:21 +00:00
Tyera Eulberg
d008dfb7ad
Bump spl-memo and spl-token versions ( #12917 )
2020-10-15 18:23:41 +00:00
behzad nouri
05cf15a382
implements DataBudget using atomics ( #12856 )
2020-10-15 11:33:58 +00:00
Ryo Onodera
a44e4d386f
Better tower logs for SwitchForkDecision and etc ( #12875 )
...
* Better tower logs for SwitchForkDecision and etc
* nits
* Update comment
2020-10-15 18:30:33 +09:00
Trent Nelson
c26512255d
RPC: Add metrics for TX encoding
2020-10-14 12:24:11 -06:00
sakridge
1f1eb9f26e
Add separate push queue to reduce push lock contention ( #12713 )
2020-10-13 18:10:25 -07:00
Michael Vines
247228ee61
Implementation-defined RPC server errors are now accessible to client/ users
2020-10-13 10:05:44 -07:00
Michael Vines
649fe6d3b6
get_vote_accounts: access HashMap directly instead of turning it into an iterator
2020-10-13 04:12:10 +00:00
Michael Vines
8a119c1483
Bump reqwest/rayon to get past `cargo audit`
2020-10-11 18:51:09 +00:00
Michael Vines
c5c8da1ac0
Expose all rewards (fees, rent, voting and staking) in RPC getConfirmedBlock and the cli
2020-10-09 21:54:13 -07:00
Ryo Onodera
1f4bcf70b0
Fix various ledger-tool error due to no builtins ( #12759 )
...
* Fix various ledger-tool error due to no builtins
* Add missing file...
2020-10-09 12:19:36 -06:00
carllin
c879e7c1ad
Fix fee mismatch on snapshot deserialize ( #12697 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-08 23:44:41 -07:00
Dan Albert
2c5f83c264
Add new internal accounts ( #12740 )
...
Co-authored-by: publish-docs.sh <maintainers@solana.com>
2020-10-09 00:48:32 +00:00
Josh
8f5431551e
Store program logs in blockstore / bigtable (TransactionWithStatusMeta) ( #12678 )
...
* introduce store program logs in blockstore / bigtable
* fix test, transaction logs created for successful transactions
* fix test for legacy bincode implementation around log_messages
* only api nodes should record logs
* truncate transaction logs to 100KB
* refactor log truncate for improved coverage
2020-10-08 12:06:15 -07:00
Michael Vines
11df2e2236
Bump version to v1.5.0
2020-10-08 04:51:36 +00:00
behzad nouri
b5faa11f73
removes invalid/outdated pending push messages early ( #12555 )
...
In CrdsGossipPush::new_push_messages:
https://github.com/solana-labs/solana/blob/972619edb/core/src/crds_gossip_push.rs#L211-L228
we already have paid the cost of looking-up the label in crds table and
checking the hash value and wallclock only to find out that in some
cases the value is invalid or is outdated. So might as well remove the
value here rather than wait for the next call to
purge_old_pending_push_messages:
https://github.com/solana-labs/solana/blob/972619edb/core/src/crds_gossip_push.rs#L372
2020-10-07 18:29:20 +00:00
Jack May
41ad3dd8f0
Remove skip-no-mangle entirely ( #12696 )
2020-10-07 09:13:48 -07:00
Trent Nelson
e35889542b
RPC: Support base64 encoded transactions
...
Defaults to base58
2020-10-06 22:41:06 -06:00
Trent Nelson
7f67d36777
RPC: Check encoded transaction size before decoding
2020-10-06 22:41:06 -06:00
behzad nouri
a5c6a78f6d
filters out inactive nodes from push options ( #12674 )
...
* filters out inactive nodes from push options
https://github.com/solana-labs/solana/pull/12620
patched the DDOS issue with nodes which go offline:
https://github.com/solana-labs/solana/issues/12409
However, offline nodes still see (much lesser) traffic spike, likely
because no origins are pruned from their bloom filter in active set:
https://github.com/solana-labs/solana/blob/aaf3790d8/core/src/crds_gossip_push.rs#L276-L286
and so multiple nodes push redundant duplicate messages to them
simultaneously:
https://github.com/solana-labs/solana/blob/aaf3790d8/core/src/crds_gossip_push.rs#L254-L255
This commit will filter out inactive peers from potential push targets
entirely. To mitigate eclipse attacks, staked nodes are retried
periodically.
* uses current timestamp in test/crds_gossip
2020-10-06 13:48:32 +00:00
Ryo Onodera
026e7de819
Show commit in `--version` and ledger-tool's log ( #12636 )
...
* Show commit in `--version` and ledger-tool's log
* Another handy hidden env var
* Fix test
* Rename to semver!
* Fix syntax error...
2020-10-04 00:30:26 +09:00
sakridge
71c469c72b
Weight push peers by how long we haven't pushed to them ( #12620 )
2020-10-02 13:57:26 -07:00
behzad nouri
2c669f65f1
limits number of threads in core/tests/crds_gossip.rs ( #12615 )
...
crds_gossip tests start large networks, which with large thread-pools
will exhaust system resources, causing failures in ci tests:
https://buildkite.com/solana-labs/solana/builds/31953
The commit limits size of thread-pools in the test.
2020-10-02 16:55:44 +00:00
Michael Vines
75b621160e
Add GetConfirmedBlocksWithLimit RPC method
2020-10-01 22:56:17 -07:00
Michael Vines
f41a73d76a
Expose validator cli arguments for pubsub buffer tuning
2020-10-01 20:30:40 -07:00
behzad nouri
1866521df6
retains hash value of outdated responses received from pull requests ( #12513 )
...
pull_response_fail_inserts has been increasing:
https://cdn.discordapp.com/attachments/478692221441409024/759096187587657778/pull_response_fail_insert.png
but for outdated values which fail to insert:
https://github.com/solana-labs/solana/blob/a5c3fc14b3/core/src/crds_gossip_pull.rs#L332-L344
https://github.com/solana-labs/solana/blob/a5c3fc14b3/core/src/crds.rs#L104-L108
are not recorded anywhere, and so the next pull request may obtain the
same redundant payload again, unnecessary taking bandwidth.
This commit holds on to the hashes of failed-inserts for a while, similar
to purged_values:
https://github.com/solana-labs/solana/blob/a5c3fc14b3/core/src/crds_gossip_pull.rs#L380
and filter them out for the next pull request:
https://github.com/solana-labs/solana/blob/a5c3fc14b3/core/src/crds_gossip_pull.rs#L204
2020-10-01 00:39:22 +00:00
Tyera Eulberg
c31a34fbcb
Include post balance information for rewards ( #12598 )
...
* Include post balance information for rewards
* Add post-balance to stored Reward struct
* Handle extended Reward in bigtable
Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-30 17:57:06 -06:00
Trent Nelson
3c7b9c2938
Move remaining nonce utils from runtime to SDK
2020-09-30 05:45:42 +00:00
behzad nouri
537bbde22e
builds crds filters in parallel ( #12360 )
...
Based on run-time profiles, the majority time of new_pull_requests is
spent building bloom filters, in hashing and bit-vec ops.
This commit builds crds filters in parallel using rayon constructs. The
added benchmark shows ~5x speedup (4-core machine, 8 threads).
2020-09-29 23:06:02 +00:00
Tyera Eulberg
96a7d4dbd8
Query BigTable for block time if does not exist in blockstore ( #12560 )
2020-09-29 21:39:36 +00:00
sakridge
ce98088457
Track inserted repair shreds ( #12455 )
2020-09-29 14:13:21 -07:00
Justin Starry
36d55c0667
Increase rpc pubsub max payload to unblock large account notifications ( #12548 )
2020-09-30 00:09:39 +08:00
behzad nouri
0d5258b6d3
separates out ClusterInfo::{gossip,listen} thread-pools ( #12535 )
...
https://github.com/solana-labs/solana/pull/12402
moved gossip-work threads:
https://github.com/solana-labs/solana/blob/afd9bfc45/core/src/cluster_info.rs#L2330-L2334
to ClusterInfo::new as a new field in the ClusterInfo struct:
https://github.com/solana-labs/solana/blob/35208c5ee/core/src/cluster_info.rs#L249
So that they can be shared between listen and gossip threads:
https://github.com/solana-labs/solana/blob/afd9bfc45/core/src/gossip_service.rs#L54-L67
However, in testing https://github.com/solana-labs/solana/pull/12360
it turned out this will cause breakage:
https://buildkite.com/solana-labs/solana/builds/31646
https://buildkite.com/solana-labs/solana/builds/31651
https://buildkite.com/solana-labs/solana/builds/31655
Whereas with separate thread pools all is good. It might be the case
that one thread is slowing down the other by exhausting the thread-pool
whereas with separate thread-pools we get fair scheduling guarantees
from the os.
This commit reverts https://github.com/solana-labs/solana/pull/12402
and instead adds separate thread-pools for listen and gossip threads:
https://github.com/solana-labs/solana/blob/afd9bfc45/core/src/gossip_service.rs#L54-L67
2020-09-29 09:05:31 +00:00
behzad nouri
57ed4e4657
patches bug in Crds::find_old_labels with pubkey specific timeout ( #12528 )
...
Current code only returns values which are expired based on the default
timeout. Example from the added unit test:
- value inserted at time 0
- pubkey specific timeout = 1
- default timeout = 3
Then at now = 2, the value is expired, but the function fails to return
the value because it compares with the default timeout.
2020-09-29 09:04:40 +00:00
Tyera Eulberg
89621adca7
Rpc -> proper optimistic confirmation ( #12514 )
...
* Add service to track the most recent optimistically confirmed bank
* Plumb service into ClusterInfoVoteListener and ReplayStage
* Clean up test
* Use OptimisticallyConfirmedBank in RPC
* Remove superfluous notifications from RpcSubscriptions
* Use crossbeam to avoid mpsc recv_timeout panic
* Review comments
* Remove superfluous last_checked_slots, but pass in OptimisticallyConfirmedBank for complete correctness
2020-09-28 20:43:05 -06:00
carllin
06f84c65f1
Fix rooted accounts cleanup, simplify locking ( #12194 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-28 16:04:46 -07:00
behzad nouri
c94fe9236f
purges old pending push messages more efficiently ( #12522 )
2020-09-28 21:59:59 +00:00
Michael Vines
31696a1d72
Port BPFLoader2 activation to FeatureSet and rework built-in program activation
2020-09-28 12:50:19 -07:00
sakridge
6583c8cffe
Add precompile verification to preflight ( #12486 )
2020-09-27 22:29:00 -07:00
Trent Nelson
1014781b87
Bump jsonrpc-* to 15.0.0
2020-09-25 19:30:47 +00:00
Michael Vines
7526bb96f3
Make test_process_rest_api less fragile
2020-09-25 11:40:36 -07:00
Michael Vines
c10da16d7b
Port instructions sysvar and secp256k1 program activation to FeatureSet
2020-09-25 11:40:36 -07:00
Michael Vines
35f5f9fc7b
Add feature set identifier to gossiped version information
2020-09-25 11:40:36 -07:00
Josh
1d04c1db94
introduce RpcPerfSample and modify getPerformanceSamples output ( #12434 )
...
* introduce RpcPerfSample and modify getPerformanceSamples output
* camelCase test results
2020-09-24 14:22:22 -07:00
behzad nouri
42f1ef8acb
moves gossip-work thread pool cons to ClusterInfo::new ( #12402 )
2020-09-24 18:36:31 +00:00
Justin Starry
6601ec8f26
Record and store invoked instructions in transaction meta ( #12311 )
...
* Record invoked instructions and store in transaction meta
* Enable cpi recording if transaction sender is some
* Rename invoked to innerInstructions
2020-09-24 22:36:22 +08:00
Justin Starry
731a943239
Remove transaction encoding from storage layer ( #12404 )
2020-09-24 13:10:29 +08:00
sakridge
68e5a2ef56
Add RPC notify and banking keys debug ( #12396 )
2020-09-23 18:46:42 -07:00
Tyera Eulberg
e1a212fb79
Bump spl-token ( #12395 )
2020-09-22 17:08:54 -06:00
Josh
65a6bfad09
Add blockstore column to store performance sampling data ( #12251 )
...
* Add blockstore column to store performance sampling data
* introduce timer and write performance metrics to blockstore
* introduce getRecentPerformanceSamples rpc
* only run on rpc nodes enabled with transaction history
* add unit tests for get_recent_performance_samples
* remove RpcResponse from rpc call
* refactor to use Instant::now and elapsed for timer
* switch to root bank and ensure not negative subraction
* Add PerfSamples to purge/compaction
* refactor to use Instant::now and elapsed for timer
* switch to root bank and ensure not negative subraction
* remove duplicate constants
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-09-22 12:26:32 -07:00
Michael Vines
208dd1de3a
Move TestValidator into its own module
2020-09-19 08:35:26 -07:00
Michael Vines
1a03afccb1
validator/ cleanup
2020-09-19 08:35:26 -07:00
Ryo Onodera
cb8661bd49
Persistent tower ( #10718 )
...
* Save/restore Tower
* Avoid unwrap()
* Rebase cleanups
* Forcibly pass test
* Correct reconcilation of votes after validator resume
* d b g
* Add more tests
* fsync and fix test
* Add test
* Fix fmt
* Debug
* Fix tests...
* save
* Clarify error message and code cleaning around it
* Move most of code out of tower save hot codepath
* Proper comment for the lack of fsync on tower
* Clean up
* Clean up
* Simpler type alias
* Manage tower-restored ancestor slots without banks
* Add comment
* Extract long code blocks...
* Add comment
* Simplify returned tuple...
* Tweak too aggresive log
* Fix typo...
* Add test
* Update comment
* Improve test to require non-empty stray restored slots
* Measure tower save and dump all tower contents
* Log adjust and add threshold related assertions
* cleanup adjust
* Properly lower stray restored slots priority...
* Rust fmt
* Fix test....
* Clarify comments a bit and add TowerError::TooNew
* Further clean-up arround TowerError
* Truly create ancestors by excluding last vote slot
* Add comment for stray_restored_slots
* Add comment for stray_restored_slots
* Use BTreeSet
* Consider root_slot into post-replay adjustment
* Tweak logging
* Add test for stray_restored_ancestors
* Reorder some code
* Better names for unit tests
* Add frozen_abi to SavedTower
* Fold long lines
* Tweak stray ancestors and too old slot history
* Re-adjust error conditon of too old slot history
* Test normal ancestors is checked before stray ones
* Fix conflict, update tests, adjust behavior a bit
* Fix test
* Address review comments
* Last touch!
* Immediately after creating cleaning pr
* Revert stray slots
* Revert comment...
* Report error as metrics
* Revert not to panic! and ignore unfixable test...
* Normalize lockouts.root_slot more strictly
* Add comments for panic! and more assertions
* Proper initialize root without vote account
* Clarify code and comments based on review feedback
* Fix rebase
* Further simplify based on assured tower root
* Reorder code for more readability
Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-19 14:03:54 +09:00
Michael Vines
c4913e3c9e
SendTransactionServices now exit their thread on channel drop instead of by a flag
2020-09-18 17:29:10 +00:00
Michael Vines
75c3690ccd
Give the duplicate send_transaction_service a different thread name
2020-09-18 17:29:10 +00:00
behzad nouri
9b866d79fb
shards crds values based on their hash prefix ( #12187 )
...
filter_crds_values checks every crds filter against every hash value:
https://github.com/solana-labs/solana/blob/ee646aa7/core/src/crds_gossip_pull.rs#L432
which can be inefficient if the filter's bit-mask only matches small
portion of the entire crds table.
This commit shards crds values into separate tables based on shard_bits
first bits of their hash prefix. Given a (mask, mask_bits) filter,
filtering crds can be done by inspecting only relevant shards.
If CrdsFilter.mask_bits <= shard_bits, then precisely only the crds
values which match (mask, mask_bits) bit pattern are traversed.
If CrdsFilter.mask_bits > shard_bits, then approximately only
1/2^shard_bits of crds values are inspected.
Benchmarking on a gce cluster of 20 nodes, I see ~10% improvement in
generate_pull_responses metric, but with larger clusters, crds table and
2^mask_bits are both larger, so the impact should be more significant.
2020-09-17 14:05:16 +00:00
Trent Nelson
32dcce0ac1
RPC: Limit request payload size to 50kB
2020-09-16 20:21:59 +00:00
Justin Starry
f6cda2579f
Fix off-by-one max payload checks
2020-09-16 12:46:06 -07:00
Michael Vines
c6eea94edc
Remove stale comment
2020-09-16 08:42:26 -07:00
Michael Vines
749208fa32
RPC sendTransaction now returns transaction logs on simulation failure
2020-09-16 08:42:26 -07:00
sakridge
3930cb865a
Add keccak-secp256k1 instruction ( #11839 )
...
* Implement keccak-secp256k1 instruction
Verifies eth addreses with ecrecover function
* Move secp256k1 test
2020-09-15 18:23:21 -07:00
Michael Vines
daae638781
Add --gossip-validator argument
2020-09-14 20:18:27 -07:00
behzad nouri
d6ec03f13c
patches default impl for crds filter ( #12199 )
...
In CrdsFilter.mask all bits after mask_bits are set to 1:
https://github.com/solana-labs/solana/blob/555252f4/core/src/crds_gossip_pull.rs#L65
However the default implementation, sets both mask and mask_bits to zero
which is inconsistent with CrdsFilter::compute_mask for a mask_bits of
zero.
This commit changes the default implementation by setting mask to
`!0u64` (i.e all bits set to one). As a result, for the default crds
filter, `test_mask` will always return true, whereas previously it was
always returning false.
https://github.com/solana-labs/solana/blob/555252f4/core/src/crds_gossip_pull.rs#L85
This is only used in tests and benchmarks, but causes some benchmarks to
be misleading by short circuiting in this line:
https://github.com/solana-labs/solana/blob/555252f4/core/src/crds_gossip_pull.rs#L429
2020-09-13 13:08:25 +00:00
Ryo Onodera
de4a613610
Check bank capitalization ( #11927 )
...
* Check bank capitalization
* Simplify and unify capitalization calculation
* Improve and add tests
* Avoid overflow and inhibit automatic restart
* Fix test
* Tweak checked sum for cap. and add tests
* Fix broken build after merge conflicts..
* Rename to ClusterType
* Rename confusing method
* Clarify comment
* Verify cap. in rent and inflation tests
Co-authored-by: Stephen Akridge <sakridge@gmail.com>
2020-09-12 01:48:06 +09:00
carllin
9c490e06b0
Fix propagation on startup from snapshot ( #12177 )
2020-09-11 02:03:11 -07:00
Michael Vines
6f325d4594
`solana-validator --rpc-bind-address` argument now works as expected
2020-09-10 13:36:13 -07:00
Michael Vines
10ce839ec0
Speed up `bigtable transaction-history` command
2020-09-10 05:22:10 +00:00
Tyera Eulberg
4431080066
Program subscriptions now properly check results len and token program id ( #12139 )
2020-09-09 21:07:29 +00:00
Tyera Eulberg
05db41fe9c
Cache block time in Blockstore ( #11955 )
...
* Add blockstore column to cache block times
* Add method to cache block time
* Add service to cache block time
* Update rpc getBlockTime to use new method, and refactor blockstore slightly
* Return block_time with confirmed block, if available
* Add measure and warning to cache-block-time
2020-09-09 09:33:14 -06:00
behzad nouri
28f2fa3fd5
uses rust intrinsics to convert hashes to u64 ( #12097 )
2020-09-09 15:28:17 +00:00
Michael Vines
9e96180ce4
getMinimumBalanceForRentExemption now only responds to valid account lengths
2020-09-08 17:48:09 +00:00
Ryo Onodera
53b8ea4464
Rename to ClusterType and restore devnet compat. ( #12068 )
...
* Rename to ClusterType and restore devnet compat.
* De-duplicate parse code and add comments
* Adjust default Devnet genesis & reduce it in tests
2020-09-08 23:55:09 +09:00
anatoly yakovenko
c67f8bd821
Forward transactions to the expected leader instead of your own TPU port ( #12004 )
...
* Use PoHRecorder to send to the right leader
* cleanup
* fmt
* clippy
* Cleanup, fix bug
Co-authored-by: Carl <carl@solana.com>
2020-09-08 17:00:49 +08:00
Justin Starry
9940870c89
Fix RPC transaction method configs serialization ( #12100 )
2020-09-08 13:08:09 +08:00
Justin Starry
369d376d10
Fix signaure subscription panic ( #12077 )
2020-09-06 14:33:12 +08:00
Ryo Onodera
eabc63cdcd
Fix bad predicate with malformed gossip votes ( #12072 )
2020-09-06 13:27:50 +09:00
Michael Vines
b64fb295a1
Add LARGEST_CONFIRMED_ROOT_UPLOAD_DELAY
2020-09-04 16:01:49 -07:00
Michael Vines
4ba43c29ce
Request correct access
2020-09-04 16:01:49 -07:00
Michael Vines
d3611f74c8
Add exit flag for bigtable upload operations
2020-09-04 16:01:49 -07:00
Michael Vines
bafdcf24f5
Add BigTableUploadService
2020-09-04 16:01:49 -07:00
Michael Vines
bc7731b969
Add BigTableUploadService
2020-09-04 16:01:49 -07:00
Michael Vines
d8e2038dda
Add --enable-bigtable-ledger-upload flag
2020-09-04 16:01:49 -07:00
behzad nouri
114c211b66
adds new CrdsFilterSet type for Vec<CrdsFilter> ( #12029 )
2020-09-04 13:04:47 +00:00
Tyera Eulberg
39246f9dd7
Revert signature-notification format change ( #12032 )
...
* Use untagged RpcSignatureResult enum to avoid breaking downstream consumers of current signature subscriptions
* Clean up client duplication
* Clippy
2020-09-03 18:14:45 -06:00
Greg Fitzgerald
2c091e4fca
Mark a withdraw authority as non-circulating ( #12033 )
2020-09-03 23:45:49 +00:00
behzad nouri
bc7adb97ed
builds crds filters without looping over filters ( #11998 )
2020-09-03 20:32:23 +00:00
Tyera Eulberg
90fad36613
Use new_response for consistency
2020-09-03 12:02:26 -06:00
Tyera Eulberg
b22de369b7
Rpc: add getMultipleAccounts endpoint ( #12005 )
...
* Add rpc endpoint to return the state of multiple accounts from the same bank
* Add docs
* Review comments: Dedupe account code, default to base64, add max const
* Add get_multiple_accounts to rpc-client
2020-09-03 11:35:06 -06:00
carllin
3f39ab1e04
Fix forwarding calculation ( #12014 )
...
Co-authored-by: Carl <carl@solana.com>
2020-09-03 02:31:28 -07:00
carllin
1c1a3f979d
Detect and notify when deserializable shreds are available ( #11816 )
...
* Add logic to check for complete data ranges
* Add RPC signature notification
Co-authored-by: Carl <carl@solana.com>
2020-09-01 22:06:06 -07:00
Michael Vines
7341e60043
Ensure that the spl-token 2 native mint account is owned by the spl-token 2 program.
...
Workaround for https://github.com/solana-labs/solana-program-library/issues/374 until spl-token 3 is shipped
2020-09-01 17:09:36 +00:00
carllin
7641b60a2b
Remove log ( #11949 )
...
Co-authored-by: Carl <carl@solana.com>
2020-08-31 16:40:22 -07:00
Tyera Eulberg
60c7ac6f95
Fix get_parsed_token_accounts ( #11907 )
2020-08-29 12:38:27 -06:00
Tyera Eulberg
befd99edac
Bump spl-token version
2020-08-29 03:20:38 -06:00
Tyera Eulberg
2eff9a19c3
Update to token pack/unpack changes ( #11900 )
2020-08-28 22:06:30 -06:00
sakridge
c4253dc0f9
Update to rayon 1.4.0 ( #11898 )
2020-08-28 21:00:11 -07:00
Tyera Eulberg
76be36c9ce
Update spl-token to v2.0 ( #11884 )
...
* Update account-decoder to spl-token v2.0
* Update transaction-status to spl-token v2.0
* Update rpc to spl-token v2.0
* Update getTokenSupply to pull from Mint directly
* Fixup to spl-token v2.0.1
2020-08-28 15:54:57 -06:00
Ryo Onodera
d8c529a9b8
Small cleaning around consensus/bank_forks ( #11873 )
2020-08-28 08:56:53 +09:00
Tyera Eulberg
5c7d743883
Timestamp first vote ( #11856 )
2020-08-26 18:34:02 +00:00
Tyera Eulberg
1988ee9cd6
Rpc: Filter accounts with invalid mints from get_parsed_token_accounts ( #11844 )
...
* Filter out accounts with invalid mints from get_parsed_token_accounts
* Explicit docs
2020-08-26 11:39:42 -06:00
Michael Vines
a1e2357d12
`solana-gossip spy` can now be given an identity keypair (`--identity` argument)
2020-08-22 17:00:50 -07:00
Michael Vines
b660704faa
Allow the sendTransaction preflight commitment level to be configured
2020-08-22 08:19:50 -07:00
behzad nouri
418b483af6
Fix filter_crds_values output alignment with the inputs ( #11734 )
2020-08-21 12:32:37 -07:00
Michael Vines
247f27af37
Submit a vote timestamp every vote ( #10630 )
...
* Submit a timestamp for every vote
* Submit at most one vote timestamp per second
* Submit a timestamp for every new vote
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-08-21 11:10:51 -06:00