Commit Graph

3162 Commits

Author SHA1 Message Date
Jeff Washington (jwash) acd846b90e
when storing detailed debug info on accounts hash, we're ok with > 1 storage (#27617) 2022-09-07 06:38:57 -07:00
Jeff Washington (jwash) ecbd5bb807
acctidx: avoid extra addref in combine ancient slots (#27545)
* acctidx: avoid extra addref in combine ancient slots

* make code common
2022-09-06 13:57:34 -07:00
Brooks Prumo a2df1e95dc
Hold `PendingAccountsPackage` lock for both checking and submitting (#27613)
Hold package lock for both checking and submitting
2022-09-06 20:11:13 +00:00
Jeff Washington (jwash) 0d01494cf8
acctidx renames for clarity (#27606) 2022-09-06 12:29:45 -07:00
Jeff Washington (jwash) f1297850b8
acctidx: use slot list len to determine if upsert remains in cache (#27550) 2022-09-06 10:59:54 -07:00
Jeff Washington (jwash) 72a2c02ee7
add --accounts-db-verify-refcounts for debugging (#27504) 2022-09-06 09:39:39 -07:00
Jeff Washington (jwash) 7bdbb18741
acctidx: factor out update_slot_list_entry (#27546) 2022-09-06 07:39:31 -07:00
Jeff Washington (jwash) 7b579771b0
log on acct idx derefs below 0 (#27544) 2022-09-06 07:34:05 -07:00
Jeff Washington (jwash) ed132e75a9
rename get_account_maps_read_lock to get_bin since we don't use a loc… (#27543)
rename get_account_maps_read_lock to get_bin since we don't ues a lock anymore
2022-09-06 07:22:55 -07:00
Jeff Washington (jwash) f90e70ff5a
fix some comments (#27547)
remove stray comment
2022-09-06 07:21:19 -07:00
Alexander Meißner 12d2147efa
Adds `IndexOfAccount` type (#27599)
Adds the type `IndexOfAccount`.
2022-09-06 11:31:40 +02:00
Alexander Meißner 6f2e556b16
Cleanup: `TransactionContext` (#27595)
* Lets instruction_accounts_lamport_sum() have the &InstructionContext as parameter directly.

* Updates docu comments.

* Uses accessors methods instead of accessing private properties of other structs.

* Adds #![deny(clippy::indexing_slicing)].

* Has get_signers() return a Result instead of using unwrap().

* Removes InvokeContext::get_key_of_account_at_index().
2022-09-05 16:29:02 +02:00
Alexander Meißner 0b94d5af18
Refactor: `InstructionContext::configure()` (#27400)
* Replaces InstructionContext::new() by InstructionContext::configure().
Adds TransactionContext::get_next_instruction_context().

* Switch back to using references as parameters.

* Hoists InstructionContext::configure() from TransactionContext::push() into InvokeContext::push().

* Moves InstructionContext::config() to the beginning of InvokeContext::push().

* Hoists InstructionContext::configure() from InvokeContext::push() into InvokeContext::process_instruction().

* Review feedback: Updates docu comments.
2022-09-03 10:34:57 +02:00
Tao Zhu c0f41fe6d0
fix a flacky test (#27572) 2022-09-02 15:02:22 +00:00
Brooks Prumo 6684c62280
Add SnapshotUsage to SnapshotConfig (#27508) 2022-09-02 08:56:23 -04:00
Tyera Eulberg 9b8bed86f9
Add getRecentPrioritizationFees RPC endpoint (#27278)
* Plumb priority_fee_cache into rpc

* Add PrioritizationFeeCache api

* Add getRecentPrioritizationFees rpc endpoint

* Use MAX_TX_ACCOUNT_LOCKS to limit input keys

* Remove unused cache apis

* Map fee data by slot, and make rpc account inputs optional

* Add priority_fee_cache to rpc test framework, and add test

* Add endpoint to jsonrpc docs

* Update docs/src/developing/clients/jsonrpc-api.md

* Update docs/src/developing/clients/jsonrpc-api.md
2022-09-01 23:12:12 +00:00
Brooks Prumo a6d737d773
Add EpochAccountsHash to AccountsDb (#27538) 2022-09-01 12:11:07 -04:00
Jeff Washington (jwash) 3e40fad46b
deserialize(or eof) snapshot epoch_accounts_hash (#27513)
deserialize(or eof) snapshot prior_epoch_accounts_hash
2022-09-01 08:35:52 -07:00
Jeff Washington (jwash) 0df0b94a72
randomly shrink ancient append vecs (#27523) 2022-09-01 08:32:43 -07:00
Haoran Yi 28c3552756 remove counter stat for clean which is already captured in clean_accounts_stats 2022-09-01 10:11:08 -05:00
Haoran Yi a28f336eb3 add timing unit scale in field name 2022-09-01 10:11:08 -05:00
Haoran Yi 43e3545c26 add stats of dirty_slots for cleaning during hash calc 2022-09-01 10:11:08 -05:00
Haoran Yi 407717eb2a add timing stat for account hash calc with cache 2022-09-01 10:11:08 -05:00
Jeff Washington (jwash) 874085aadc
improve performance of unrefing ancient pubkeys (#27485) 2022-09-01 07:34:52 -07:00
apfitzge af971e63ab
Create all shared buffer readers before spawning threads (#27512)
* Create all shared buffer readers before spawning threads

* Allow 'needless' collect - clippy is wrong

* move clippy allow, spacing
2022-08-31 22:49:02 -05:00
Alexander Meißner 84b5a2bcf4
Refactor: `TransactionContext` ABIv2 adjustments (#27476)
* Marks methods and structs which are only to be used by the program runtime.

* Removes TransactionContext::get_keys_of_accounts().

* Adds InstructionContext::get_instruction_accounts_lamport_sum().
2022-08-31 17:47:47 +02:00
Jeff Washington (jwash) 8c1e193d5d
delay trying to flush cached upserts until far future (#26908) 2022-08-31 08:56:26 -05:00
Tao Zhu 8bb039d08d
collect min prioritization fees when replaying sanitized transactions (#26709)
* Collect blocks' minimum prioritization fees when replaying sanitized transactions

* Limits block min-fee metrics reporting to top 10 writable accounts

* Add service thread to asynchronously update and finalize prioritization fee cache

* Add bench test for prioritization_fee_cache

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2022-08-31 08:00:55 -05:00
Jeff Washington (jwash) b5cca0bfa3
add tests to AccountsDb::unref_accounts (#27474) 2022-08-30 18:51:20 -05:00
Jeff Washington (jwash) 3d03f7b47e
remove unused acct idx::upsert_on_disk (#27479) 2022-08-30 15:42:46 -05:00
Jeff Washington (jwash) 2e4a52d010
remove storages that only contain empty (#27452) 2022-08-30 14:04:17 -05:00
Jeff Washington (jwash) 1154bc8636
get rid of acquisition of read lock while unrefing accounts in scan (#27388) 2022-08-30 14:04:02 -05:00
Jeff Washington (jwash) 9bde3261fd
cleanup (#27453)
* cleanup

* Update runtime/src/accounts_db.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-08-30 10:58:35 -05:00
Jeff Washington (jwash) e35cd7a34a
comment loop in function (#27454) 2022-08-30 10:58:02 -05:00
Jeff Washington (jwash) 3d692ba046
refactor clean_dead_slots_from_accounts_index (#27451) 2022-08-30 10:06:52 -05:00
Jeff Washington (jwash) 73ee94d674
remove old bank hashes with ancient shrink (#27450) 2022-08-30 09:31:20 -05:00
Jeff Washington (jwash) a4b8ab2f59
clean_dead_slots_from_accounts_index unrefs correctly (#27461) 2022-08-30 08:40:21 -05:00
Haoran Yi 8f3c1e3ebb report a datapoint 2022-08-29 19:06:30 -05:00
Haoran Yi 7e722f15f2 reduce log frequence to 1m when qlen is above the threshold 2022-08-29 19:06:30 -05:00
Haoran Yi 37bba64613 remove bank drop queue stat
remove try_send
add warn! for excessively long bank drop queue
2022-08-29 19:06:30 -05:00
Haoran Yi 5b64107626 make pruned_bank channel unbonded.
In kin-sim, we found that bounded channel causes halt for account
background services. As the number of accounts grows, the time for
pruning and cleaning increases, which would leads to longer intervals
between the pruning of deaded bank slots. With 1.7B accounts, we will
exceed the 10K bounded channel threshold that causes halt of account
back ground services. Without pruning, the node will eventually run out
of memory.
2022-08-29 19:06:30 -05:00
Brooks Prumo 757e46c3c7
Set cap for new allocations per transaction (#27385) 2022-08-29 14:30:48 -04:00
apfitzge 49411aaae0
Feature: Interleaved Snapshot Untar and Indexing - Stage 1 (#26590)
* Add snapshot_storage_lengths_from_fields

* Add SnapshotUnpacker

* Add SnapshotStorageRebuilder

* Rebuild snapshot storage during unpack

* move snapshot unpacker back into snapshot_utils

* use SerializedAppendVecId in storage rebuilder

* create AccountStorageMap type alias

* Wrap storage and next_append_vec_id in a type

* fixed typo

* move use StorageAndNextAppendVecId

* calculate num_rebuilder_threads more concisely
2022-08-29 13:17:27 -05:00
Brooks Prumo 3c7cd62030
Move pruned_banks_receiver into PrunedBanksRequestHandler (#27445) 2022-08-29 13:30:06 -04:00
Michael Vines d0983c3cf7 Update runtime background hash verifier thread name 2022-08-26 13:44:11 -07:00
Trent Nelson 05fe061a7c Address RUSTSEC-2022-0051
`lz4` to 1.24.0
`lz4-sys` to 1.9.4
2022-08-26 14:30:47 -06:00
Brennan Watt 46a48760db
Switch concurrent replay from feature to param (#27401)
* Switch concurrent replay from feature to param
2022-08-26 12:36:02 -07:00
Jeff Washington (jwash) cdb699d812
ancient append vecs and cleaning dead slots (#27353)
cleanup old code
2022-08-26 10:13:41 -07:00
Tao Zhu 5e71f339c9
Add a SanitizedMessage type that caches writable accounts indexes (#27317)
* Add a SanitizedMessage type that caches writable accounts indexes

* Add is_writable_account_cache to both SanitizedMessage variants, cache is initialized in constructors
2022-08-25 16:33:41 -05:00
Jeff Washington (jwash) 9866510ef2
get rid of extra comment (#27378) 2022-08-24 14:10:33 -05:00