a18debc34a
allow test feature to skip rewrites ( #33851 )
...
* allow test feature to skip rewrites
* hook up cli arg for test skip rewrites, update tests
* fix sanity checker
* add account hash to abi to fix a test
* reviews
* use hashmap to collect skip_rewrites. exclude skip_rewrites from dirty
pubkey set
* accumulate skipped_rewrite in reduce
* mutex
* fmt
* skip hash verify for this test flag
* add skipped rewrites num stat
* skip bank hash verify not account hash verify
* reviews
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-27 09:14:05 -05:00
510b6b949f
Split compute budget instructions process from struct ( #33852 )
...
* Split compute budget instruction processing from ComputeBudget struct itself, so CB instructions can be processed elsewhere without involving ComputeBudget
* updated tests
* avoid built ComputeBudget from dated ComputeBudgetLimits in this refactoring PR
* Clean-up program-runtime/src/compute_budget_processor.rs
* Add test for a corner case that deprecated instruction is used to request units greater than max limit;
* Update code to handle the corner case.
2023-10-26 22:12:56 -05:00
bd1080b26f
Adds AtomicAge to bucket map holder ( #33841 )
2023-10-25 10:47:21 -04:00
af7fd32f4c
chore: fix some typos ( #33833 )
...
* fix spelling of "retrieved"
* fix spelling of "should"
* fix spelling of "comparisons"
2023-10-25 16:21:53 +02:00
612e8e8457
Removes Default and pub from CumulativeOffset ( #33840 )
2023-10-24 19:22:34 +00:00
abf51864b4
Removes `#[allow(dead_code)]` in StorableAccounts impl ( #33821 )
2023-10-24 18:31:27 +00:00
9064b8b2a2
Optimize account hash CumulativeOffset index from vec to 2-element array ( #33839 )
...
* optimize account hash merkle tree index from vec to 2-element array
* typo
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-24 18:26:16 +00:00
2f024f0cc1
Uses AccountHash in CalculateHashIntermediate ( #33822 )
2023-10-24 08:39:32 -04:00
b0b4e1f0c0
remove IncludeSlotInHash after feature activation on mnb ( #33816 )
...
* remove IncludeSlotInHash after feature activation on mnb
* fix compile errors
* compile errors
* fix tests
* fix test results
2023-10-23 15:12:02 -07:00
54b796f5a1
ancient pack: add low water mark ( #33785 )
2023-10-23 10:56:18 -07:00
669bc43bcd
Buffer account's fields for hash ( #33788 )
...
* buffer accounts field for hash
* use smallvec to allocate hash buffer on stack
* sort deps
* more opt
* clippy
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-23 09:48:17 -05:00
a41b24f185
fix some ancient stats ( #33791 )
2023-10-23 07:23:47 -07:00
34103e0913
typo in variable name ( #33801 )
2023-10-23 07:23:11 -07:00
e13756133a
ancient: add many_refs_this_is_newest_alive ( #33741 )
...
* add many_refs_this_is_newest_alive
* fix pathological case, add comments
* add log
* update comments
* add log when we fail to pack
* use with_capacity
* fix log comment
* clippy
2023-10-20 09:55:37 -07:00
c98c24bd6d
Revert "Split compute budget instructions process from struct itself … ( #33784 )
...
Revert "Split compute budget instructions process from struct itself (#33513 )"
This reverts commit c73bebe984
2023-10-20 15:53:45 +02:00
ce8ad77373
Uses AccountHash in AppendVec ( #33764 )
2023-10-19 17:38:09 -04:00
f13c78b7c8
add ancient append vec test ( #33762 )
...
* add ancient append vec test
* Update accounts-db/src/ancient_append_vecs.rs
Co-authored-by: Brooks <brooks@prumo.org>
* add some comments
---------
Co-authored-by: Brooks <brooks@prumo.org>
2023-10-19 09:50:38 -07:00
c73bebe984
Split compute budget instructions process from struct itself ( #33513 )
...
* Split compute budget instruction processing from ComputeBudget struct itself, allow compute_budget_instructions be processed elsewhere without having to instantiate ComputeBudget
* updated tests
2023-10-19 11:10:42 -05:00
4e5c545e23
Uses AccountHash in tiered storage ( #33763 )
2023-10-19 11:52:36 -04:00
0fcc0a0941
Returns AccountHash from get_filler_account() ( #33758 )
2023-10-18 20:29:40 -04:00
1045548606
Uses AccountHash in StorableAccountsWithHashesAndWriteVersions ( #33751 )
2023-10-18 18:58:19 -04:00
3a580f4df6
StorableAccounts::hash() returns &AccountHash ( #33748 )
2023-10-18 15:41:26 -04:00
94273434ce
[TieredStorage] HotStorageReader::get_account_meta_from_offset() ( #33724 )
...
#### Problem
HotStorageReader currently only implements get_footer(). It does not
have a function to obtain the account meta.
#### Summary of Changes
This PR implements HotStorageReader::get_account_meta_from_offset().
A function that returns the account meta located at the specified offset.
This will be the helper function that will be later used to obtain the account
meta when the offset is available from the index block of a hot storage file.
#### Test Plan
A new test is included in this PR.
2023-10-18 12:27:39 -07:00
e96678b302
Uses SeqLock for CachedAccountInner::hash ( #33696 )
2023-10-18 10:43:35 -04:00
0b05e8db11
[TieredStorage] Footer test for HotStorageReader ( #33718 )
...
#### Problem
HotStorageReader currently doesn't have a test that covers its footer.
#### Summary of Changes
This PR includes a test for HotStorageReader that verifies the footer.
2023-10-17 15:24:38 -07:00
c09cbbb778
sort ancient append vec target_slots_sorted ( #33729 )
2023-10-17 14:49:46 -07:00
56a7485998
log ancient stats each time we run ( #33730 )
2023-10-17 14:41:23 -07:00
673a38c892
move timer after early exit ( #33732 )
...
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-17 15:20:29 -05:00
f178975185
typo ( #33734 )
...
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-17 14:45:17 -05:00
5de9163625
add metric for ancient can't move slots ( #33713 )
...
* add metric for ancient can't move slots
* rename
* fix erors in replacing text
* rename
2023-10-17 08:34:18 -07:00
d948e5bf69
ancient shrink on its own cadence ( #33712 )
2023-10-16 10:06:20 -07:00
167dac204f
Retry hash file allocation ( #33565 )
...
* retry hash file allocation
* add sleep
* submit a datapoint for retry
* typo
* more typos
* Update accounts-db/src/accounts_hash.rs
Co-authored-by: Brooks <brooks@prumo.org>
* fmt
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
Co-authored-by: Brooks <brooks@prumo.org>
2023-10-16 09:21:08 -05:00
c354879a3f
stop padding new append vecs to page size ( #33658 )
...
* stop padding new append vecs to page size
* fix test
* fix another test
* for creating test accounts, allocate larger like we used to
2023-10-12 07:11:20 -07:00
1a2c7f106e
fix typo in comments ( #33665 )
...
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-12 09:05:10 -05:00
982d29cf60
Revert "stop padding new append vecs to page size ( #33607 )" ( #33634 )
...
This reverts commit b7962a3610
2023-10-10 14:43:52 -05:00
cb695c7b32
Cleanup - Feature Gate of `enable_early_verification_of_account_modifications` ( #33579 )
...
Cleans up the feature gate of enable_early_verification_of_account_modifications:
- Removes PreAccount
- Removes InvokeContext::pre_accounts and InvokeContext::rent
- Removes InvokeContext::verify() and InvokeContext::verify_and_update()
- Removes TransactionContext::is_early_verification_of_account_modifications_enabled()
- Removes TransactionAccounts::is_early_verification_of_account_modifications_enabled
- No longer optional: TransactionContext::rent
2023-10-10 17:26:17 +02:00
b7962a3610
stop padding new append vecs to page size ( #33607 )
...
* stop padding new append vecs to page size
* for creating test accounts, allocate larger like we used to
2023-10-10 07:03:37 -07:00
6c7d3c8aeb
remove redundant page_align ( #33608 )
2023-10-10 07:02:47 -07:00
fc73813db2
Adds AccountHash newtype ( #33597 )
2023-10-09 16:00:52 -04:00
052677595c
in hash calc, delete old cache files that will not be used earlier ( #33432 )
...
* in hash calc, delete old cache files that will not be used earlier
* only delete if supposed to
* fmt
2023-10-09 11:47:39 -07:00
c924719040
Removes hash param from AccountsCache::store() ( #33598 )
2023-10-09 18:16:15 +00:00
72574dac02
Assert acount hash mmap file capacity > 0 ( #33575 )
...
assert mmap capacity > 0
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-09 09:38:00 -05:00
bb27bd88d4
Removes write version from tiered storage ( #33566 )
2023-10-06 16:19:35 -04:00
ecb1f8a9d7
[TieredStorage] Include Hot Account in StoredAccountMeta and ReadableAccount ( #33544 )
...
#### Problem
All account storage formats are required to implement both StoredAccountMeta
and ReadableAccount, but the implementation for the hot account format is missing.
#### Summary of Changes
This PR includes hot account format into StoredAccountMeta and ReadableAccount
enum. This will allow the TieredStorageReader in the future PRs to return hot account
format in its `get_account` implementation.
2023-10-06 11:54:02 -07:00
c66af12bdb
Uses `IntSet` for `RemoveUnrootedSlotsSynchronization::slots_under_contention` ( #33530 )
2023-10-04 16:34:18 -04:00
3b93041401
Uses `IntSet` for dead slots ( #33529 )
2023-10-04 16:21:44 -04:00
befc903993
Uses `IntSet` in AccountsDb::calc_delete_dependencies() ( #33528 )
2023-10-04 16:21:06 -04:00
f714a44c2a
Uses `IntSet` for `RollingBitField::excess` ( #33523 )
2023-10-04 15:04:28 -04:00
5a9956824f
Uses `IntSet` for `RootsTracker::uncleaned_roots` ( #33524 )
2023-10-04 13:26:31 -04:00
d41fa346cc
Uses `IntSet` for `ShrinkCandidates` ( #33522 )
2023-10-04 13:07:35 -04:00
bb19ebed49
Convert tuple into dedup result struct ( #33450 )
...
convert tupe into dedup result struct
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-29 21:12:50 -05:00
0e9e91c65e
Drops the shrink_candidate_slots lock after done inserting ( #33459 )
2023-09-29 16:58:55 +00:00
b81ff5d654
Fixup the metrics for remove_dead_accounts_shrink_us ( #33458 )
2023-09-29 16:04:33 +00:00
6ea51280dd
Move sort timer out of loop ( #33448 )
...
move sort timer out of loop
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-28 20:10:41 -05:00
25c27d452c
hash dedup vec ( #33246 )
...
* hash_dedup vec algo
* reviews
* reviews
* more reviews
* simplify working_set init with add_next_item
* refactor to remove special case "new" from add_item.
The new change is that, even the new item is the new min, it will still be added to working_set.
This change will make init working_set code simpler and the loop loop check simpler.
Since the item is inserted in at the end of the vector, the cost of push into and pop from the working will be O(1), shouldn't affect performance much.
* comments
* refactor unnamed tuple in working set to SlotGroupPointer type
* use SlotGroupPointer in ItemLocation
* Add Copy traits to avoid explicty call of clone on SlotGroupPointer
* consume next in add_next_item fn (credit to jeff).
note that the old code is still correct, since before call to
add_next_item, we will have already overwritten `next` to correct value.
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-28 08:58:08 -05:00
e088eb2be0
Code clean up ( #33417 )
...
clean up
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-26 22:35:25 +00:00
746f69772a
[TieredStorage] Streamline the handling of TieredStorageFormat ( #33396 )
...
#### Problem
The TieredStorageFormat field in the TieredStorage is only used in the write path.
#### Summary of Changes
This PR simplifies the handling of TieredStorageFormat by removing its field from
TieredStorage struct but passing via write_accounts().
2023-09-26 14:05:36 -07:00
9f6f532535
`flush_slot_cache_with_clean()` takes a single Slot ( #33413 )
2023-09-26 15:38:37 -04:00
a9b0fb492b
split hash calc ancient slot boundary to allow for ancient shrinking … ( #33216 )
...
split hash calc ancient slot boundary to allow for ancient shrinking to be behind
2023-09-26 07:27:50 -07:00
23ad476ffb
Removes unused `Versioned` trait ( #33360 )
2023-09-25 17:18:38 -04:00
40f536010f
visit_duplicate_pubkeys_during_startup uses scan ( #33397 )
2023-09-25 12:48:29 -07:00
18231e9a5a
dump final startup index stats only after startup is complete ( #33400 )
2023-09-25 12:19:52 -07:00
1b15464414
diskidx: stats for created vs reused ( #33385 )
2023-09-23 06:22:12 -07:00
7cf71011fd
disk bucket: init restart path ( #33375 )
2023-09-22 11:40:46 -07:00
7c545b0ae9
bump rust stable to 1.72.1 ( #33333 )
...
* bump rust stable to 1.72.1
* bump rust nightly to 2023-09-20
* fix nightly lint -- unused doc comment
-- rustdoc does not generate documentation for expression fields
* fix nightly lint -- unnecessarily eager cloning of iterator items
* fix nightly lint -- loop never actually loops
2023-09-20 23:53:36 -06:00
c85eb73300
O(n) dedup ( #33297 )
...
O(N) dedup
Co-authored-by: jeff washington <jeff.washington@solana.com>
2023-09-18 17:55:28 -05:00
8b8a21a52f
cleanup feature: enable request heap frame instruction #30076 ( #33243 )
...
* cleanup feature: enable request heap frame instruction #30076
* update sbf tests
* removed out dated comments and test
2023-09-18 16:06:24 -05:00
17c3930bc8
in gen index, stop using big hashmap ( #33252 )
...
* in gen index, stop using big hashmap
* update accounts_data_len
* remove approx_stored_count
2023-09-18 11:13:18 -07:00
402981e3c1
cleanup test ( #33291 )
2023-09-18 10:59:42 -07:00
27caf4d1d2
add test for duplicates in generate_index and fix approx stored count ( #33290 )
...
add test
2023-09-18 09:46:52 -07:00
9e11ae6275
Make program owners a const array instead of Vec<_> ( #33275 )
...
The program owners pubkeys are constant, no need to reconstruct the
Vec<Pubkey> and Vec<&Pubkey> each time this function runs (every time we
execute transactions).
2023-09-18 10:59:03 -05:00
6283c1d568
Refactors out `unsafe` from cache_hash_data.rs ( #33271 )
2023-09-15 14:25:43 -04:00
a47f65d882
Refactors `unsafe` out of CacheHashDataFile's header ( #33270 )
2023-09-15 16:49:35 +00:00
f77b3d9389
Derives Pod for CalculateHashIntermediate ( #33267 )
2023-09-15 12:47:38 -04:00
4c42413c1f
Refactors out `unsafe` from MmapAccountHashesFile::read() ( #33266 )
2023-09-15 15:09:01 +00:00
3ad8394047
stop removing empty bins during index generation ( #33242 )
2023-09-14 07:53:33 -07:00
886eabd74d
gen idx: refactor StorageSizeAndCount population ( #33244 )
2023-09-14 07:53:05 -07:00
602f0f6ecb
generate index: fn arg: &Slot -> Slot ( #33229 )
...
fn arg: &Slot -> Slot
2023-09-13 14:14:30 -07:00
c2bf2a969d
remove unused metrics ( #33241 )
2023-09-13 14:14:10 -07:00
cb310a310a
set approx store count at generate index time ( #33240 )
2023-09-13 14:13:46 -07:00
093ae61e85
single pass through each storage to populate info ( #33227 )
2023-09-13 12:37:01 -07:00
3ffd78f2d7
add comments ( #33226 )
2023-09-12 13:51:27 -07:00
21e7f98b3a
Uses hex for the hash in accounts hash cache file names ( #33196 )
2023-09-12 11:26:08 -04:00
c61ee20b44
Transient accounts hash cache dir is unnecessary ( #33181 )
2023-09-12 11:23:07 -04:00
3cd3994939
clean up pub crate ( #33214 )
...
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-11 19:28:26 -05:00
09936aac0e
Removes `new()` from POD CalculateHashIntermediate and CumulativeOffset ( #33211 )
2023-09-11 22:33:25 +00:00
cf35799b2a
remove unused stats ( #33213 )
...
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-11 21:03:37 +00:00
659fc6837b
Removes `pub` from accounts_hash.rs items ( #33209 )
2023-09-11 20:49:51 +00:00
d724af863c
add a few accounts hash dedup tests ( #33208 )
2023-09-11 13:20:28 -07:00
6298c6c31e
Shares accounts hash cache data between full and incremental ( #33164 )
2023-09-11 15:55:48 -04:00
bbb57be0a5
Refactor - Move interfaces of address-lookup-table into the program SDK ( #33165 )
...
* Adds a module `address_lookup_table` to the SDK.
* Adds a module `address_lookup_table::instruction` to the SDK.
* Adds a module `address_lookup_table::error` to the SDK.
* Adds a module `address_lookup_table::state` to the SDK.
* Moves AddressLookupTable into SDK as well.
* Moves AddressLookupTableAccount into address_lookup_table.
* Adds deprecation messages.
* Disentangles dependencies across cargo files.
2023-09-11 21:10:40 +02:00
4dfe62a2f0
rework accounts hash calc dedup to avoid kernel file error ( #33195 )
...
* in hash calc, calculate max_inclusive_num_pubkeys
* in hash calc, dedup uses mmap files to avoid os panic
* as_mut_ptr
* remove unsafe code
* refactor count in hash files
---------
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-11 10:57:21 -07:00
a145ade564
stat for time spent copying generate index contents ( #33187 )
...
* stat for time spent copying generate index contents
* rework to move stat to generate_index
* fix fmt
2023-09-08 12:55:54 -07:00
dc6b1eb653
in shrink, get stores prior to shrink starting ( #33194 )
2023-09-08 08:45:32 -07:00
0083e426b2
ShrinkCandidates only holds slot ( #33173 )
2023-09-07 12:28:05 -07:00
e331275178
remove coupling of ShrinkCandidates to HashMap ( #33176 )
2023-09-07 09:07:37 -07:00
f4816dc0cf
Moves CacheHashData test-only methods into tests module ( #33170 )
2023-09-06 15:25:58 -04:00
a3dc3eb37c
Takes inner HashSet when dropping CacheHashData ( #33169 )
2023-09-06 18:19:01 +00:00
88ee8f5820
replaces once_cell::sync::OnceCell with std::sync::OnceLock ( #33140 )
...
std::sync::OnceLock has become stable since rust 1.70.0 and there is no
longer a need for an external crate dependency.
2023-09-06 16:46:51 +00:00
a8bc6ebe38
Use `write_all` for binned account hash file writes ( #33095 )
...
write_all
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-06 11:03:05 -05:00
Jeff Washington (jwash)
Tao Zhu
Brooks
Kevin Heavey
HaoranYi
steviez
Yueh-Hsuan Chiang
Alexander Meißner
Trent Nelson
behzad nouri