Commit Graph

2320 Commits

Author SHA1 Message Date
Tyera Eulberg bb9d2fd07a
Cli: use get_inflation_rewards and limit epochs queried (#16408)
* Fix block-with-limit when not finalized blocks found

* Enable confirmed commitment in getInflationReward

* Use get_inflation_rewards in cli

* Line up rewards output

* Add range validator

* Change cli epoch arg -> num epochs

* Add solana inflation rewards subcommand

* Consolidate epoch rewards meta
2021-04-08 10:57:33 -06:00
Josh e501fa5f0b
Rpc: introduce get_inflation_reward rpc call (#16278)
* feat: introduce get_inflation_reward rpc call

* fix: style suggestions

* fix: more style changes and match how other rpc functions are defined

* feat: get reward for a single epoch

* feat: default to the most recent epoch

* fix: don't factor out get_confirmed_block

* style: introduce from impl for RpcEncodingConfigWrapper

* style: bring commitment into variable

* feat: support multiple pubkeys for get_inflation_reward

* feat: add get_inflation_reward to rpc client

* feat: return rewards in order

* fix: rename pubkeys to addresses

* docs: introduce jsonrpc docs for get_inflation_reward

* style: early return in map (not sure which is more idiomatic)

* fix: call the rpc client function args addresses as well

* fix: style

* fix: filter out only addresses we care about

* style: make this more idiomatic

* fix: change rpc client epoch to optional and include some docs edits

* feat: filter out rent rewards in get_inflation_reward

* feat: add option epoch config param to get_inflation_reward

* feat: rpc client get_inflation_reward takes epoch instead of config and some filter staking and voting rewards
2021-04-06 18:10:53 -07:00
carllin 1219842a96
No wallclock throttle tests (#16396) 2021-04-05 19:40:16 -07:00
Trent Nelson b71875df61 cluster-info: Get rid of some integer math while we're here 2021-04-06 00:09:37 +00:00
Trent Nelson b6b08706b9 cluster-info: Don't subtract non-shred spies from node count 2021-04-06 00:09:37 +00:00
Trent Nelson 7a2a39093d validator: Use a const for wait for supermajority threshold 2021-04-05 17:29:37 -06:00
Michael Vines 43feef7362 Adjust tokio version to just "1" 2021-04-05 17:13:20 +00:00
Michael Vines b242f82696 Reduce test-validator ledger size 2021-04-05 08:37:29 -07:00
behzad nouri 701fc93343
patches bug in banking stage where buffered packets are never retained (#16276)
banking_stage::handle_forwarding is retaining buffered packets with
empty index, so nothing is held:
https://github.com/solana-labs/solana/blob/6f3926b64/core/src/banking_stage.rs#L520
2021-04-05 12:46:21 +00:00
sakridge 3429785d9b
Wait for 90 percent of stake before starting (#16340) 2021-04-03 14:21:20 -07:00
carllin 4e5ef6bce2
Add cluster state verifier logging (#16330)
* Add cluster state verifier logging

* Add duplicate-slots iterator to ledger tool
2021-04-02 21:48:44 -07:00
Tyera Eulberg da27acabcc
Rpc: enable getConfirmedSignaturesForAddress2 to return confirmed (not yet finalized) data (#16281)
* Update blockstore method to allow return of unfinalized signature

* Support confirmed sigs in getConfirmedSignaturesForAddress2

* Add deprecated comments

* Update docs

* Enable confirmed transaction-history in cli

* Return real confirmation_status; fill in not-yet-finalized block time if possible
2021-04-01 04:35:57 +00:00
Tyera Eulberg 18bd47dbe1
Rpc: fix getConfirmedTransaction slot (#16288)
* Fix transaction blockstore apis

* Update blockstore apis in rpc
2021-03-31 21:04:00 -06:00
behzad nouri 3f63ed9a72
removes OrderedIterator and transaction batch iteration order (#16153)
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817

However load_and_execute_transactions is iterating over
  lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.

If iteratorion_order is [1, 2, 0], and i is 0, then:
  lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
  transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).

This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.
2021-03-31 23:59:19 +00:00
sakridge 54c68ea83f
Drop write lock on sysvars (#15497)
* Drop write lock on sysvars

* adds env var for demoting sysvar write lock demotion

* moves demote logic to is_writable

* feature gates sysvar write lock demotion

* adds builtins to write lock demotion

* adds system program id to builtins

* adds Feature111...

* adds an abi-freeze test

* mvines set of builtin program keys

Co-authored-by: Michael Vines <mvines@gmail.com>

* update tests

* adds bpf loader keys

* Add test sysvar

* Plumb demote_sysvar to is_writable

* more plumbing of demote_sysvar_write_locks to is_writable

* patches test_program_bpf_instruction_introspection

* hard codes demote_sysvar_write_locks to false for serialization/encoding methods

* Revert "hard codes demote_sysvar_write_locks to false for serialization/encoding methods"

This reverts commit ae3e2d2e777437bddd753933097a210dcbc1b1fc.

* change the hardcoded ones to demote_sysvar_write_locks=true

* Use data_as_mut_slice

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-03-30 10:05:09 -07:00
Jeff Washington (jwash) 414c7070cb
poll checking for new record in poh service after every batch of hashes instead of busy waiting (#16167)
* poll waiting in poh service after every batch of hashes

* clippy
2021-03-30 10:34:21 -05:00
Jeff Washington (jwash) 5eff23db0c
TransactionRecorder uses unique channel so we can use Recv instead of RecvTimeout (#16195)
* time

* new channel each call

* new channel every time
2021-03-30 00:51:35 -05:00
Tyera Eulberg 3977ed5c82
Future-aware enum name 2021-03-29 14:58:05 -06:00
Tyera Eulberg 60ed8e2892
Rpc: enable getConfirmedBlocks and getConfirmedBlocksWithLimit to return confirmed (not yet finalized) data (#16161)
* Add commitment config capabilities

* Use rpc limit if no end_slot provided

* Limit to actually finalized blocks

* Support confirmed blocks in getConfirmedBlocks and getConfirmedBlocksWithLimit

* Update docs

* Add client plumbing

* Rename config enum
2021-03-29 12:41:31 -06:00
sakridge 60b4771fc6
Only print skipped leader slot message when the node is actually leader (#16156)
Also, check vote signature after the vote is signed
2021-03-26 17:45:53 -07:00
Tyera Eulberg 433f1ead1c
Rpc: enable getConfirmedBlock and getConfirmedTransaction to return confirmed (not yet finalized) data (#16142)
* Add Blockstore block and tx apis that allow unrooted responses

* Add TransactionStatusMessage, and send on bank freeze; also refactor TransactionStatusSender

* Track highest slot with tx-status writes complete

* Rename and unpub fn

* Add commitment to GetConfirmed input configs

* Support confirmed blocks in getConfirmedBlock

* Support confirmed txs in getConfirmedTransaction

* Update sigs-for-addr2 comment

* Enable confirmed block in cli

* Enable confirmed transaction in cli

* Review comments

* Rename blockstore method
2021-03-26 16:47:35 -06:00
Jeff Washington (jwash) 4f4cffbd03
Throttle PoH ticks by cumulative slot time (#16139)
* Throttle PoH ticks by cumulative slot time

* respond to pr feedback

* saturating sub

* updated comment
2021-03-26 18:54:16 +00:00
Jeff Washington (jwash) 06ac0fe9a3
increase timeout in TransactionRecorder.record (#16133) 2021-03-25 21:31:07 -05:00
sakridge b99ae8f334
Skip leader slots until a vote lands (#15607) 2021-03-25 18:54:51 -07:00
behzad nouri b041b55028
makes test_pull_request_time_pruning smaller (#16128) 2021-03-25 22:44:43 +00:00
sakridge 9b94741290
Fix test_replay_commitment_cache (#16131) 2021-03-25 21:16:39 +00:00
Justin Starry e817a6db00
Add timeout for local cluster partition tests (#16123)
* Add timeout for local cluster partition tests

* fix optimistic conf test logs

* Bump instruction count assertions
2021-03-25 13:27:07 -06:00
carllin 52703badfa
Setup ReplayStage confirmation scaffolding for duplicate slots (#9698) 2021-03-24 23:41:52 -07:00
Tyera Eulberg a8ef29df27
Support getBlockTime for unfinalized blocks (#16103) 2021-03-24 20:52:08 -06:00
sakridge 96ccc40f0a
Set ticks_per_slot higher for banking stage tests (#16094)
Tests are timing out because the bank hit the MaxTickHeight and
will not process the transactions.
2021-03-24 14:05:43 -07:00
Jeff Washington (jwash) f68860a643
poh record metrics (#16092) 2021-03-24 14:48:32 -05:00
behzad nouri a6c23648cb
limits CrdsGossipPull::pull_request_time size (#15793)
There is no pruning logic on CrdsGossipPull::pull_request_time
https://github.com/solana-labs/solana/blob/79ac1997d/core/src/crds_gossip_pull.rs#L172-L174
potentially allowing this to take too much memory.

Additionally, CrdsGossipPush::last_pushed_to is pruning recent push
timestamps:
https://github.com/solana-labs/solana/blob/79ac1997d/core/src/crds_gossip_push.rs#L275-L279
instead of the older ones.

Co-authored-by: Nathan Hawkins <utsl@utsl.org>
2021-03-24 18:33:56 +00:00
behzad nouri 570fd3f810
makes turbine peer computation consistent between broadcast and retransmit (#14910)
get_broadcast_peers is using tvu_peers:
https://github.com/solana-labs/solana/blob/84e52b606/core/src/broadcast_stage.rs#L362-L370
which is potentially inconsistent with retransmit_peers:
https://github.com/solana-labs/solana/blob/84e52b606/core/src/cluster_info.rs#L1332-L1345

Also, the leader does not include its own contact-info when broadcasting
shreds:
https://github.com/solana-labs/solana/blob/84e52b606/core/src/cluster_info.rs#L1324
but on the retransmit side, slot leader is removed only _after_ neighbors and
children are computed:
https://github.com/solana-labs/solana/blob/84e52b606/core/src/retransmit_stage.rs#L383-L384
So the turbine broadcast tree is different between the two stages.

This commit:
* Removes retransmit_peers. Broadcast and retransmit stages will use tvu_peers
  consistently.
* Retransmit stage removes slot leader _before_ computing children and
  neighbors.
2021-03-24 13:34:48 +00:00
Justin Starry e7fd7d46cf
rpc: add getSlotLeaders method (#16057) 2021-03-23 17:48:54 +00:00
behzad nouri 4f82b897bc
buffers data shreds to make larger erasure coded sets (#15849)
Broadcast stage batches up to 8 entries:
https://github.com/solana-labs/solana/blob/79280b304/core/src/broadcast_stage/broadcast_utils.rs#L26-L29
which will be serialized into some number of shreds and chunked into FEC
sets of at most 32 shreds each:
https://github.com/solana-labs/solana/blob/79280b304/ledger/src/shred.rs#L576-L597
So depending on the size of entries, FEC sets can be small, which may
aggravate loss rate.
For example 16 FEC sets of 2:2 data/code shreds each have higher loss
rate than one 32:32 set.

This commit broadcasts data shreds immediately, but also buffers them
until it has a batch of 32 data shreds, at which point 32 coding shreds
are generated and broadcasted.
2021-03-23 14:52:38 +00:00
Jeff Washington (jwash) 57ba86c821
eliminate lock on record (#15929)
* eliminate lock on record

* use same error as MaxHeightReached

* clippy

* review feedback

* refactor should_tick code

* pr feedback
2021-03-23 09:10:04 -05:00
Jeff Washington (jwash) 2fc609a294
add metric for ticks from poh_recorder.record (#16047) 2021-03-22 15:35:06 -05:00
Tyera Eulberg 2ec24d438f
Make getStakeActivation response consistent for undelegated accounts (#16038) 2021-03-19 14:54:56 -06:00
Jeff Washington (jwash) ddc758439e
metrics for poh_recorder.record (#15998) 2021-03-19 09:48:55 -05:00
Michael Vines 02b81dd05d Update to reqwest 0.11.2 2021-03-18 11:59:41 -07:00
Tyera Eulberg aa54c468ea
rpc: Add config options limiting getConfirmedBlock response data (#15970)
* Add new confirmed block struct

* Add RpcConfirmedBlockConfig options

* Configure block response based on new options

* Add client api, use in cli fetch_epoch_rewards

* Update docs

* Apply review suggestions
2021-03-18 17:58:20 +00:00
Michael Vines 04c99cf7ea Add --slots-per-epoch argument 2021-03-17 22:56:41 +00:00
carllin f548a04fae
Allow unbounded wallclock processing time in tests (#15961) 2021-03-17 15:48:50 -07:00
Michael Vines 59c19d9fbf Notice the user when the --mint, --bpf-program, or --clone arguments are ignored 2021-03-17 20:04:53 +00:00
Michael Vines 8a9b51952e Ignore flaky test_banking_stage_entries_only and test_banking_stage_entryfication 2021-03-17 11:28:56 -07:00
Jeff Washington (jwash) 40997d0aef
add metrics for tick producer and poh_recorder (#15931) 2021-03-17 10:38:26 -05:00
Jeff Washington (jwash) 5460fb10a2
drop poh lock after record (#15930) 2021-03-17 10:37:20 -05:00
Michael Vines a2eb655322 =1.7.0 2021-03-16 07:51:07 +00:00
Jeff Washington (jwash) efee8b62d7
a few missed set_data calls (#15846)
* a few missed set_data calls

* another set data call
2021-03-15 21:57:23 -05:00
Jeff Washington (jwash) c09ea2c314
More AccountSharedData construction (#15844)
* one more AccountSharedData construction

* one more construct
2021-03-15 19:27:17 -05:00
carllin c1ba265dd9
Wallclock BankingStage Throttle (#15731) 2021-03-15 17:11:15 -07:00
Michael Vines 0c9ca5522c Bump version to v1.7.0 2021-03-13 09:01:21 +00:00
Tyera Eulberg 5b2da19c93
Rpc: support extended config for getConfirmedBlock (#15827)
* Add rpc confirmed-block config wrapper to support struct of extended config

* Update docs

* Make config wrapper generic and use in getConfirmedTransaction as well

* Update/clean confirmed-tx docs
2021-03-12 22:19:45 +00:00
behzad nouri f2865dfd63
requires stakes for propagating crds values through gossip (#15561) 2021-03-12 15:50:14 +00:00
Justin Starry 918d04e3f0
Add more slot update notifications (#15734)
* Add more slot update notifications

* fix merge

* Address feedback and add integration test

* switch to datapoint

* remove unused shred method

* fix clippy

* new thread for rpc completed slots

* remove extra constant

* fixes

* rely on channel closing

* fix check
2021-03-12 21:44:06 +08:00
Ryo Onodera 4bbeb9c033
Remove old feature: simple_capitalization (#15763)
* Remove old feature: simple_capitalization

* Fix another failing test in core

* Finish up test cleanup

* Further clean up a bit
2021-03-12 11:12:40 +09:00
Jeff Washington (jwash) 952c3bcbb7
AccountSharedData construction (#15790) 2021-03-11 18:09:04 -06:00
Trent Nelson 24d18b3cf2 docs: add docs links for crates published to crates.io 2021-03-11 08:38:18 +00:00
Jeff Washington (jwash) 1135ffd595
mut data refs as slice (#15782) 2021-03-10 15:28:03 -06:00
behzad nouri 56923c91bf
limits number of unique pubkeys in the crds table (#15539) 2021-03-10 20:46:05 +00:00
Jeff Washington (jwash) 52e54e1100
account.data -> data() (#15778) 2021-03-09 22:31:33 +00:00
Jeff Washington (jwash) 8a3135d17b
Account->AccountSharedData (#15691) 2021-03-09 15:06:07 -06:00
carllin 2bee9435f3
Add tracer key for tracing transaction path through the network (#15732) 2021-03-08 19:31:00 -08:00
carllin 331c45decf
Report datapoint on number of retransmit shreds (#15694) 2021-03-08 17:54:53 -08:00
sakridge d09112fa6d
PoH batch size calibration (#15717) 2021-03-05 16:01:21 -08:00
Michael Vines 4a3ab77baf Remove unused id field 2021-03-05 19:07:59 +00:00
Michael Vines 66b781eec3 Add 'unknown' health check state 2021-03-05 17:46:50 +00:00
Tyera Eulberg 7e65289729
Convert blockstore TransactionStatus column family to protobufs (#15733)
* Prevent panic if TransactionStatus can't be deserialized

* Convert Blockstore TransactionStatus column to protobuf

* Add compatability test
2021-03-05 09:05:35 -07:00
Michael Vines bd13262b42 Add validator startup process reporting before RPC is available 2021-03-05 08:03:36 -08:00
Michael Vines 24ab84936e Break up RPC API into three categories: minimal, full and admin 2021-03-04 16:39:44 -08:00
Jeff Washington (jwash) 34bebb7d09
report execution details in replay time (#15693) 2021-03-04 11:38:12 -06:00
Jeff Washington (jwash) be35c1c1b7
add execute detail timings (#15638) 2021-03-03 17:07:45 -06:00
behzad nouri 658951e680
sends only the latest vote of each validator to the banking stage (#15629) 2021-03-03 19:07:16 +00:00
carllin aacb28c453
Only report metrics every second (#15652) 2021-03-03 10:58:47 -08:00
sakridge 830be855dc
Forward and hold packets (#15634) 2021-03-03 10:23:05 -08:00
Tyera Eulberg 19ac79b5cc
Deprecate UiTokenAmount::ui_amount (#15616)
* Add TokenAmount::ui_amount_string

* Fixup solana-tokens

* Update docs
2021-03-02 22:51:41 -07:00
Tyera Eulberg a4f0033bd7
Remove ValidatorConfig derive Clone, and fix local-cluster tests (#15647)
* Remove ValidatorConfig derive Clone

* Add local-cluster ValidatorConfig helpers

* Fix benches
2021-03-03 04:21:30 +00:00
behzad nouri 0bd0084b0d
adds more metrics for tx counts and batch sizes (#15642) 2021-03-03 01:28:15 +00:00
behzad nouri 416ea38028
adds metrics for the size and number of batches in bank_send_loop (#15627) 2021-03-02 15:44:35 +00:00
Greg Fitzgerald 2463cc1e6a
Fix typos (#15610) 2021-03-02 06:36:49 -08:00
Michael Vines 640e36287e Move ValidatorExit into ValidatorConfig, making it accessible from the solana-validator crate 2021-03-01 16:49:56 -08:00
sakridge f1223fb783
Lower blockstore processor error severity (#15578) 2021-03-01 14:57:37 -08:00
Trent Nelson ef63ffce7d Bump ed25519-dalek to 1.0.1 2021-03-01 21:10:46 +00:00
carllin ae96ba3459
Plumb slot update pubsub notifications (#15488) 2021-02-28 23:29:11 -08:00
behzad nouri f7a049f87f
coalesces vote packets into one Packets (#15566) 2021-02-26 23:23:08 +00:00
sakridge 05409e51ce
Increase tpu coalescing and add parameter (#15536)
Should create larger entries on average
2021-02-26 09:15:45 -08:00
behzad nouri 5a9896706c
indexes epoch slots in crds table (#15459)
ClusterInfo::get_epoch_slots_since scans the entire crds table to obtain
epoch-slots inserted since a timestamp:
https://github.com/solana-labs/solana/blob/013daa8f4/core/src/cluster_info.rs#L1245-L1262
The alternative is to index epoch-slots in crds table ordered by their
insert timestamp.
2021-02-26 14:12:04 +00:00
Tyera Eulberg 1ad2c9f741
Revert "Make UiTokenAmount::ui_amount a String (#15447)" (#15542)
This reverts commit d14374bc9f.
2021-02-25 21:53:40 +00:00
Michael Vines 5b54aed1c0 Speed up getLeaderSchedule 2021-02-24 11:17:25 -08:00
Justin Starry 61ed980ac0
Fix received notifications for gossip signature subscriptions (#15506) 2021-02-24 16:59:22 +08:00
carllin c2e8814dce
Add limit and shrink policy for recycler (#15320) 2021-02-24 00:15:58 -08:00
Tyera Eulberg 52f2d425e5
Count if optimistically confirmed slot is already rooted (#15492) 2021-02-23 22:03:22 +00:00
sakridge 1b59b163dd
Add max retransmit and shred insert slot (#15475) 2021-02-23 13:06:33 -08:00
Michael Vines 4b0114b991 Limit the number of getProgramAccounts filters 2021-02-23 18:43:22 +00:00
Tyera Eulberg d14374bc9f
Make UiTokenAmount::ui_amount a String (#15447)
* Make UiTokenAmount::ui_amount a String

* Fixup solana-tokens

* Ignore spl downstream-project
2021-02-22 13:05:45 -07:00
Ryo Onodera 5ccaa6336a
Print original error from accounts dir remove (#15458) 2021-02-22 21:24:09 +09:00
Ivan Mironov 013daa8f47 RPC: Improve snapshot path sanitization 2021-02-20 13:06:07 -08:00
Michael Vines 5df36aec7d Pacify clippy 2021-02-19 20:08:41 -08:00
Michael Vines fd3b71a2c6 cargo fmt 2021-02-19 20:08:41 -08:00
behzad nouri aa3aac766f
adds metrics for inbound/outbound gossip packets counts (#15407) 2021-02-19 22:49:35 +00:00
Justin Starry 15bbe6436d
Bump SPL token version to v3.1.0 (#15429)
* Bump SPL token version to v3.1.0

* Cargo.lock
2021-02-19 09:15:41 +00:00
Tyera Eulberg 170cb792eb
Return blockstore error if previous_blockhash cannot be determined (#15382)
* Return blockstore error if previous_blockhash cannot be determined

* Add require_previous_blockshash flag
2021-02-18 01:04:52 +00:00
Trent Nelson 7f7370c306 Re-allow clippy::integer_arithmetic at crate-level 2021-02-17 13:55:08 -07:00
Michael Vines 9ba69a7381 Adapt to fs_extra 1.2.0 2021-02-17 12:44:58 -08:00
sakridge b24cb9840e
Speedup ledger cleanup test (#15304)
Just clone to produce shreds and use a separate insert thread.
2021-02-17 08:59:25 -08:00
Tyera Eulberg 98e3e570d2 Add --force arg for bigtable upload 2021-02-16 17:24:07 -08:00
Jeff Washington (jwash) ba02452d75
add validator flag no-accounts-db-index-hashing (#15350)
* add validator flag no_accounts_db_index_hashing

* add validator flag no_accounts_db_index_hashing
2021-02-16 21:13:48 +00:00
behzad nouri f79c9d4094
adds an upper bound on cluster-slots size (#15300)
https://github.com/solana-labs/solana/issues/14366#issuecomment-769096305
2021-02-16 21:12:13 +00:00
behzad nouri 076c20f1ca
checks that prune-messages have the same inner/outer pubkey (#15352) 2021-02-16 21:06:18 +00:00
Trent Nelson b3256aa722 Bump rand_core to 0.6.2
https://rustsec.org/advisories/RUSTSEC-2021-0023
2021-02-15 22:51:35 +00:00
sakridge 5b8f046c67
More configurable rocksdb compaction (#15213)
rocksdb compaction can cause long stalls, so
make it more configurable to try and reduce those stalls
and also to coordinate between multiple nodes to not induce
stall at the same time.
2021-02-14 10:16:30 -08:00
carllin 629dcd0f39
Cleanup buffered packets (#15210) 2021-02-12 03:27:37 -08:00
carllin 990bb426a9
Fix flaky test test_concurrent_snapshot_packaging (#15252) 2021-02-11 16:03:51 -08:00
Josh 4013f91dbe
RPC: add caching to getLargestAccounts (#15154)
* introduce get largest accounts cache

* remove cache size and change hash key

* remove eq and hash derivation from commitment config

* add slot to the cache
2021-02-11 11:32:46 -08:00
Tyera Eulberg 948819dfa8
Use spl-token-mint secondary index for relevant getProgramAccounts requests (#15219) 2021-02-09 15:49:13 -07:00
Tyera Eulberg da6753b8c0
Warp timestamp and extend max-allowable-drift for accommodate slow blocks (#15204)
* Remove timestamp_correction feature gating

* Remove timestamp_bounding feature gating

* Remove unused deprecated ledger code

* Remove unused deprecated unbounded-timestamp code

* Enable independent adjustment of fast/slow timestamp bounding

* Update timestamp bounds to 25% fast, 80% slow; warp timestamp

* Update bank hash test

* Add PR number to feature

Co-authored-by: Michael Vines <mvines@gmail.com>

Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-09 15:49:00 -07:00
behzad nouri 2758588ddd
uses btree-map instead of hash-map for cluster-slots (#15194)
retain traverses all values in the hashmap which is slow:
https://github.com/solana-labs/solana/blob/88f22c360/core/src/cluster_slots.rs#L45
btree-map instead allows more efficient prunning there.

In addition there is potential race condition here:
https://github.com/solana-labs/solana/blob/88f22c360/core/src/cluster_slots.rs#L68-L74
If another thread inserts a value at the same slot key between the read
and write lock, current thread will discard the inserted value.
2021-02-09 22:04:41 +00:00
Michael Vines f063f02c41 solana-test-validator now uses the BPF JIT by default, `--no-bpf-jit` to disable 2021-02-09 20:24:14 +00:00
Michael Vines 2caed6bd4c Complete --rpc-pubsub-max-in-buffer-capacity/--rpc-pubsub-max-out-buffer-capacity plumbing 2021-02-09 07:24:30 +00:00
behzad nouri b6f231b60e
removes locked pubkey references (#15152) 2021-02-08 02:07:00 +00:00
sakridge bbae23358c
ledger-tool cleanup and additions (#15179)
* Plumb allow-dead-slots to ledger-tool verify

* ledger-tool cleanup and add some useful missing args

Print root slots and how many unrooted past last root.
2021-02-06 17:26:42 -08:00
Jeff Washington (jwash) fbf9dc47e9
Threadpool2 (#15151)
* rework thread pool for hash calculation

* rename
2021-02-05 18:48:16 -06:00
Jeff Washington (jwash) fabecdc86c
use thread pool for non-index hash calculations (#15149) 2021-02-05 19:48:55 +00:00
behzad nouri 6fd5ec0e4c
caches descendants in bank forks (#15107) 2021-02-05 18:00:45 +00:00
Ryo Onodera c69027e5aa
chore: bump serde from 1.0.118 to 1.0.122 (#15126)
* chore: bump serde from 1.0.118 to 1.0.122

* Update programs/bpf/Cargo.lock
2021-02-05 19:23:24 +09:00
Tyera Eulberg d1563f0ccd
Bump tonic, prost, tarpc, tokio (#15013)
* Update tonic & prost, and regenerate proto

* Reignore doc code

* Revert pull #14367, but pin tokio to v0.2 for jsonrpc

* Bump backoff and goauth -> and therefore tokio

* Bump tokio in faucet, net-utils

* Bump remaining tokio, plus tarpc
2021-02-05 00:21:53 -07:00
Jeff Washington (jwash) 600ff0d915
calculate hash from store instead of index (#15034)
* calculate hash from store instead of index

* restore update hash in abs
2021-02-04 09:00:33 -06:00
behzad nouri 86467d825a
removes pubkey references (#15050) 2021-02-03 23:02:11 +00:00
behzad nouri 0ad063f4e9
adds flag to disable duplicate instance check (#15006) 2021-02-03 16:26:17 +00:00
Tyera Eulberg 98aa1fa4ea
Upgrade jsonrpc crates to v17.0.0 (#15018)
* Upgrade to jsonrpc 17.0.0

* Fix test

* tree

Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-02 19:53:08 -07:00
dependabot[bot] 1df93fa2be
chore: bump serde from 1.0.112 to 1.0.118 (#14828)
* chore: bump serde from 1.0.112 to 1.0.122

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.112 to 1.0.122.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.122)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* Update frozen_abi digest following serde update

* Revert "chore: bump serde from 1.0.112 to 1.0.122"

This reverts commit a3ef4442a4c985144ae2bd7ceaf8899a7ab8d7c0.

* Revert "[auto-commit] Update all Cargo lock files"

This reverts commit c41c3b005fb1ccade55155302c52cd5736c4b55f.

* chore: bump serde from 1.0.112 to 1.0.118

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.112 to 1.0.118.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.118)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* Remove serum-dex pinning

* blind commit!

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-02-02 23:28:16 +09:00
Tyera Eulberg cbb8b79a60
Add validator flag to opt in to cpi and logs storage (#14922)
* Add validator flag to opt in to cpi and logs storage

* Default TestValidator to opt-in; allow using in multinode-demo

* No clone

Co-authored-by: Carl Lin <carl@solana.com>
2021-02-01 14:00:51 -07:00
Tom Parker-Shemilt 01230a0105
Remove serial_test_derive dependency (#14891) 2021-01-28 22:35:31 -07:00
Ryo Onodera d6873b82ab
Remove potentially too costly Packets::default() (#14821)
* Remove potentially too costly Packets::default()

* Fix test...

* Restore Packets::default()

* Restore Packets::default() more
2021-01-29 09:32:38 +09:00
Michael Vines 119e2c75dd cli now supports a custodian for stake authorize operations 2021-01-26 11:48:28 -08:00
Tyera Eulberg ffa5c7dcc8
Deprecate commitment variants (#14797)
* Deprecate commitment variants

* Add new CommitmentConfig builders

* Add helpers to avoid allowing deprecated variants

* Remove deprecated transaction-status code

* Include new commitment variants in runtime commitment; allow deprecated as long as old variants persist

* Remove deprecated banks code

* Remove deprecated variants in core; allow deprecated in rpc/rpc-subscriptions for now

* Heavier hand with rpc/rpc-subscription commitment

* Remove deprecated variants from local-cluster

* Remove deprecated variants from various tools

* Remove deprecated variants from validator

* Update docs

* Remove deprecated client code

* Add new variants to cli; remove deprecated variants as possible

* Don't send new commitment variants to old clusters

* Retain deprecated method in test_validator_saves_tower

* Fix clippy matches! suggestion for BPF solana-sdk legacy compile test

* Refactor node version check to handle commitment variants and transaction encoding

* Hide deprecated variants from cli help

* Add cli App comments
2021-01-26 19:23:07 +00:00
behzad nouri d1df9da7d3
fixes test_filter_current flakiness (#14816) 2021-01-25 15:57:46 +00:00
Ryo Onodera bd0433c373
Clean up VerifiedVotePackets (#14822) 2021-01-25 09:01:47 +00:00
behzad nouri e1021d9f83
removes redundant epoch stakes cache in retransmit (#14781)
Following d6d76219b, staked nodes computed from vote accounts are
already cached in runtime::Stakes, so the caching in retransmit_stage is
redundant.
2021-01-24 21:15:09 +00:00
behzad nouri 491b059755
broadcasts duplicate shreds through gossip (#14699) 2021-01-24 15:47:43 +00:00
Michael Vines cbffab7850 Upgrade to Rust v1.49.0 2021-01-23 19:16:36 -08:00
Michael Vines bf1943e489 Add solana-test-validator --warp-slot argument 2021-01-22 21:17:02 -08:00
Michael Vines cbb9ac19b9 Add ability to clone accounts from an RPC endpoint 2021-01-22 13:29:36 -08:00
Tyera Eulberg 71e9958e06
Rpc: Add custom error for BigTable data not found (#14762)
* Expose not-found bigtable error

* Add custom rpc error for bigtable data not found

* Return custom rpc error when bigtable block is not found

* Generalize long-term storage
2021-01-22 04:40:47 +00:00
behzad nouri e4da6761a7
fixes test_filter_current flakiness (#14749) 2021-01-21 21:53:10 +00:00
behzad nouri 8e581601d6
patches crds vote-index assignment bug (#14438)
If tower is full, old votes are evicted from the front of the deque:
https://github.com/solana-labs/solana/blob/2074e407c/programs/vote/src/vote_state/mod.rs#L367-L373
whereas recent votes if expire are evicted from the back:
https://github.com/solana-labs/solana/blob/2074e407c/programs/vote/src/vote_state/mod.rs#L529-L537

As a result, from a single tower_index scalar, we cannot infer which crds-vote
should be overwritten:
https://github.com/solana-labs/solana/blob/2074e407c/core/src/crds_value.rs#L576

In addition there is an off by one bug in the existing code. tower_index is
bounded by MAX_LOCKOUT_HISTORY - 1:
https://github.com/solana-labs/solana/blob/2074e407c/core/src/consensus.rs#L382
So, it is at most 30, whereas MAX_VOTES is 32:
https://github.com/solana-labs/solana/blob/2074e407c/core/src/crds_value.rs#L29
Which means that this branch is never taken:
https://github.com/solana-labs/solana/blob/2074e407c/core/src/crds_value.rs#L590-L593
so crds table alwasys keeps 29 **oldest** votes by wallclock, and then
only overrides the 30st one each time. (i.e a tally of only two most
recent votes).
2021-01-21 13:08:07 +00:00
Michael Vines 04ce33a04e Ensure sanitary transactions 2021-01-20 23:59:32 -08:00
Ryo Onodera dcaa025822
Configure Bigtable's timeout, enabling by default (#14657)
* Configure bigtable's timeout when read-only

* Review comments

* Apply nits (thanks!)

Co-authored-by: Michael Vines <mvines@gmail.com>

* Timeout in the streamed decoding as well

Co-authored-by: Michael Vines <mvines@gmail.com>
2021-01-19 13:57:16 +00:00
behzad nouri b5fd0ed859
rewrites turbine retransmit peers computation (#14584) 2021-01-19 04:18:47 +00:00
behzad nouri c6ae0667e6
feature gates turbine retransmit peers patch (#14631) 2021-01-19 04:16:19 +00:00