Commit Graph

22004 Commits

Author SHA1 Message Date
Tyera cbbae8fcd4
Update accounts-cluster-bench to work against public clusters (#30781)
* Move client ctor outside start_sig_clear_thread

* Add TransactionExecutor::new_with_url method and dedupe

* Remove unnecessary pub

* Update airdrop to use RpcClient

* Construct client outside run_accounts_bench

* Add config and url parameters
2023-03-20 09:46:06 -06:00
behzad nouri 4de59881b7
filters out merkle shreds until feature activation (#30769)
In order to maintain backward compatibility, the commit reworks merkle
shreds feature gate to off by default until the feature activation.
2023-03-20 15:44:00 +00:00
behzad nouri 4b595ebaaf
adds metrics tracking deduper saturations (#30779) 2023-03-20 15:33:36 +00:00
Jeff Washington (jwash) df2c95119c
add buckets stats for # files and total file size (#30780)
* add buckets stats for # files and total file size

* change metrics to bytes

* get index file size correct
2023-03-20 08:22:31 -07:00
bji d77f0a22c7
Add new vote state version that replaces Lockout with LandedVote to a… (#29524)
* Add new vote state version that replaces Lockout with LandedVote to allow vote latency to be tracked in a future change. Includes a feature to be enabled which will when enabled cause the vote state to be written in the new form.

* Update feature set key to one owned by ashwin

---------

Co-authored-by: Ashwin Sekar <ashwin@solana.com>
2023-03-20 08:31:46 -06:00
behzad nouri 46614c0e9f
removes false_positive_rate field from Deduper (#30788)
removes the false_positive_rate field from the Deduper

Deduper.false_positive_rate field is misleading because it is not
enforced until maybe_reset is called. But then maybe_reset can be
invoked with an explicit argument.
2023-03-20 13:16:52 +00:00
Brooks cd7fe76744
Removes writing BankIncrementalSnapshotPersistence in AccountsHashVerifier (#30792) 2023-03-19 21:45:13 -04:00
Brooks 989c80342b
Adds plumbing and tests for verify_snapshot_bank() with Incremental Accounts Hash (#30789) 2023-03-19 21:44:38 -04:00
Illia Bobyr ea7c72f7a8
ledger: confirm_slot_entries(): Refactor: reduce indent (#30785)
No functional changes.  Use early return pattern, to reduce indentation
in the most important part of the confirmation logic.
2023-03-18 19:25:18 -07:00
Brooks d3273ba118
Removes unused `ancestors` param from `get_snapshot_storages` (#30773) 2023-03-18 17:17:41 -04:00
Brooks 8325b0f591
Do not use ancestors in get_snapshot_storages (#30772)
* Do not use ancestors in get_snapshot_storages

* fixup verify_accounts_hash bench
2023-03-17 22:48:47 +00:00
Xiang Zhu 8e3a30c22c
Clean orphaned account snapshot dirs (#30645)
* Clean up orphaned account snapshot hardlink dirs

* fix compilation issues

* debugged, now working.  seeing the orphaned directories deleted

* change back to eprintln + exit for account_path error

* changed eprintln to panic for now

* add test_clean_orphaned_account_snapshot_dirs for codecov check

* address a few comments and nit isseus

* directly unzip, skipped the intermediate array of tuples

* let set_up_account_run_and_snapshot_paths return Result

* 'proper' typo, and comment on return

* use map_err

* use for loop in clean_orphaned_account_snapshot_dirs, removed panic

* add test_set_up_account_run_and_snapshot_paths

* minor, replace .for_each with .all

* rename set_up_account_run_and_snapshot_paths to create_all_accounts_run_and_snapshot_dirs

* remove unnecessary closure return type

* change to for loop

* change match to unwrap_or_else

* remove create_dir_all(&account_path) in create_all

* minor comment cleanup
2023-03-17 15:22:10 -07:00
Illia Bobyr c449a15c30
metrics: datapoint macro: Allow trailing comma (#30317)
Rust grammar allows trailing commas in most places where a list of
elements are accepted.  It simplifies cases when the list is generated
by a macro, allowing the macro to avoid special cases for a one element
list vs longer lists.

As such, it is a common practice to allow trailing commas in macros as
well.
2023-03-17 12:57:46 -07:00
Jeff Washington (jwash) 6dd5a22926
when writing to disk bucket index, tune towards packing tighter (#30761)
* when writing to disk bucket index, tune towards packing tighter

* switch to min
2023-03-17 12:34:56 -07:00
Illia Bobyr 1a71cd7fe2
measure: Measure::end_as{ns,us,ms,s,duration} (#30762)
In most cases, for a given `Measure` value, a `stop()` call must always be followed by a call to one of the `as_*()` methods. Combining them into a single function call makes the API a bit simpler.
2023-03-17 12:26:06 -07:00
Brooks d4dd4360e7
Removes `base_slot` param from `calculate_incremental_accounts_hash()` (#30774) 2023-03-17 15:22:22 -04:00
Brooks af367db6f0
Calculates accounts hash from storages in snapshot tests (#30778) 2023-03-17 15:22:02 -04:00
Ryo Onodera 05ee06814e
Fix wrongly swapped parent slot and block height (#30744) 2023-03-17 20:07:52 +09:00
Tyera 20223b2557
Revert ledger-tool changes, fix master (#30763)
Revert 10f49d4 for ledger-tool
2023-03-16 19:30:10 -06:00
steviez f1cd64f4a1
ledger-tool: chore: Cleanup snapshot retention constants (#30745)
Declare constants at top of function with comment, and stringify the
constants when needed to be in line with how other defaults are done in
ledger-tool main function.
2023-03-17 09:25:58 +08:00
behzad nouri 93f696dac7
increases shred-fetch-stage deduper capacity and reset-cycle (#30690) 2023-03-17 00:05:29 +00:00
cavemanloverboy 10f49d4e26
Geyser Runtime Reload (#30352)
Support dynamic geyser plugin load, unload, and listing through admin RPC.
2023-03-16 17:03:00 -07:00
Tyera 9d792c1848
Ledger-tool: add param to deactivate features in snapshot creation (#30755)
* Add separate parameter for deactivating features on snapshot creation

* Add debug logs for account/feature removals

* Add the word gate
2023-03-16 15:54:05 -06:00
Jeff Washington (jwash) d66d1f7a46
disk index handles empty slot list more correctly (#30752) 2023-03-16 12:34:45 -07:00
behzad nouri 7a7b020580
dedups packets using an atomic bloom filter (#30726)
Current Deduper implementation uses many bits per entry:
https://github.com/solana-labs/solana/blob/65cd55261/perf/src/deduper.rs#L70-L73
and may be saturated quickly. It also lacks api to specify desired false
positive rate.

The commit instead uses an atomic bloom filter with K hash functions.
The false positive rate is obtained by tracking popcount of bits.
2023-03-16 16:45:42 +00:00
Tao Zhu 17ba051e1f
Revert "add default_cost as mandatory field for Builtin (#30639)" (#30749)
This reverts commit 7b95c8e698.
2023-03-16 11:21:06 -05:00
ryleung-solana 0ed9f62602
Quic server batching (#30330) 2023-03-16 21:50:57 +08:00
Brooks 8be99c1c7d
Verifies incremental accounts hash in verify_accounts_hash_and_lamports (#30735) 2023-03-16 09:48:06 -04:00
Brooks 6bdbd2dfec
Removes unnecessary AccountsHashVerifier from snapshot tests (#30738) 2023-03-16 09:17:42 -04:00
Ryo Onodera aaaee556d8
ci: silence ci test output while recording in full (#30654)
* ci: silence ci test output while recording in full

* shellcheck

* Adjust --color handling place

* Dump to stderr...

* Reduce too spammy solana_metrics logs

* Clean up

* Tweak

* Stash actual command's exit_code, not echo's
2023-03-16 22:17:29 +09:00
Brooks 423fd6010e
Removes extraneous accounts hash calculations in snapshot tests (#30737) 2023-03-16 09:17:03 -04:00
Tyera 0cdb315b5b
Make redelegate signer more optional (#30734) 2023-03-15 22:59:56 -06:00
Yihau Chen f8abc8df8d
ci: fix do-audit don't report error (#30728)
* ci: fix do-audit don't retport error

* Update ci/do-audit.sh

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* use the latest suggestion

* lint

* redirect stderr to stdout

* another solution

---------

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2023-03-16 11:58:08 +08:00
Yueh-Hsuan Chiang fce949009d
Add new_from_file() API to AccountsFile (#30687)
#### Problem
Accounts db currently use AppendVec::new_from_file() directly
to create a new AcountsFile instance from an existing file.
However, this method should be abstracted out to AccountsFile
so that an existing file can be opened correctly using the right format.

#### Summary of Changes
Add new_from_file() API to AccountsFile which will open an existing
file based on its accounts file format.

Currently, it only supports AppendVec.
2023-03-15 20:38:20 -07:00
Andrew Fitzgerald b7e76c752f
Separate stats updates from decision_maker (#30481)
* Separate stats updates from decision_maker

* BufferedPacketsDecision::bank_start

* BufferedPacketsDecision: bank_start() doc-comment

* remove unnecessary clone
2023-03-15 19:39:48 -07:00
Ryo Onodera 74970a0b5d
Remove unused ProcessOptions::entry_callback (#30600)
* Confine entry_callback under cfg(test) for clarity

* Fix ci

* Actually remove entry_callback altogether

* fix clippy
2023-03-16 09:33:18 +09:00
behzad nouri 3cc74a6087
updates tempfile to 3.4.0 (#30727)
Patching below rustsec advisory:

    Crate:     remove_dir_all
    Version:   0.5.3
    Title:     Race Condition Enabling Link Following and Time-of-check Time-of-use (TOCTOU)
    Date:      2023-02-24
    ID:        RUSTSEC-2023-0018
    URL:       https://rustsec.org/advisories/RUSTSEC-2023-0018
    Solution:  Upgrade to >=0.8.0
2023-03-15 18:25:35 -06:00
Brooks cbc0d01d0d
Refactors verify_accounts_hash_and_lamports() (#30733) 2023-03-15 18:37:27 -04:00
samkim-crypto 2d58bb287d
[zk-token-sdk] Add option to create proof context state in the proof verification program (#29996)
* extend verifiable trait

* add PodBool

* implement ZkProofData trait

* add proof context program to zk-token-proof program

* update tests  for close account

* add close account instruction

* reorganize tests

* complete tests

* clean up and add docs

* clean up pod

* add proof program state

* update tests

* move proof program tests as separate module

* clippy

* cargo sort

* cargo fmt

* re-organize visibility

* add context state description

* update maintainer reference

* change `VerifyProofData` and `ProofContextState` to pod

* add tests for mixing proof types

* add tests for self owned context state accounts

* cargo fmt

* remove unnecessary scoping and add comments on scopes

* re-organize proof instructions

* clippy

* update zk-token-proof-test to 1.16.0

* upgrade spl-token-2022 to 0.6.1

* reoganize proof type

* cargo lock

* remove ZkProofContext trait
2023-03-16 07:35:20 +09:00
Tao Zhu a4ad0c75fc
fix heap cost calculation rounding error (#30673)
* add test for refaction heap size, fix size truncating by div op
* add feature gate
* checked result of consume_checked() if feature is activated
2023-03-15 12:03:04 -05:00
Brooks cefb00e3fb
Includes error in panic in initial accounts hash verification (#30725) 2023-03-15 12:24:55 -04:00
Brooks a5f86a8212
Verifies accounts hash in snapshot tests (#30724) 2023-03-15 12:23:44 -04:00
Brooks 65cd55261a
Makes Bank::get_snapshot_hash() aware of incremental accounts hashes (#30719) 2023-03-14 18:03:38 -04:00
Jeff Washington (jwash) 62fe6ea7a0
validators always skip clean/shrink on startup (#30710)
* validators always skip clean/shrink on startup

* move arg to get_deprecated_arguments
2023-03-14 11:41:15 -07:00
Nick Frostbutter f528335ac0
docs: prioritization fees (#30138)
* docs: updated prioritization fees docs

* fix: updated wording and removed outdated info

* docs: updated prioritization fees docs

* fix: merge fix

---------

Co-authored-by: nickfrosty <nick.frostbutter@solana.org>
2023-03-14 13:31:41 -05:00
Tyera b389d509a8
Track max_complete_rewards_slot for use in rpc, bigtable (#30698)
* Add RewardsMessage enum

* Cache and update max_complete_rewards_slot

* Plumb max_complete_rewards_slot into JsonRpcRequestProcesseor

* Use max_complete_rewards_slot to check get_block requests

* Use max_complete_rewards_slot to limit Bigtable uploads

* Plumb max_complete_rewards_slot into RpcSubscriptions

* Use max_complete_rewards_slot to limit block subscriptions

* Nit: fix test
2023-03-14 12:08:48 -06:00
Brooks 93c43610ac
AccountsHashVerifier stores IncrementalAccountsHash in AccountsDb (#30696) 2023-03-14 12:41:44 -04:00
Brennan 9b587bf073
Create gossip vote iterator sorted by stake weight (#30697) 2023-03-14 08:43:01 -07:00
Brennan 11c942ab40
`test_verified_vote_packets_validator_gossip_votes_iterator_correct_fork` dynamic num validators (#30695)
Gossip vote test dynamic number of validators cleanup
2023-03-14 08:35:26 -07:00
Jeff Washington (jwash) 89d5efa0d8
mem idx tracks duplicates better (#30701) 2023-03-14 07:26:39 -07:00