Commit Graph

145 Commits

Author SHA1 Message Date
HaoranYi 669bc43bcd
Buffer account's fields for hash (#33788)
* buffer accounts field for hash

* use smallvec to allocate hash buffer on stack

* sort deps

* more opt

* clippy

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-23 09:48:17 -05:00
Jeff Washington (jwash) a41b24f185
fix some ancient stats (#33791) 2023-10-23 07:23:47 -07:00
Jeff Washington (jwash) 34103e0913
typo in variable name (#33801) 2023-10-23 07:23:11 -07:00
Tao Zhu af9c754690
Crates have identical build.rs to frozen-abi can just be symlink (#33787)
crates have identical build.rs to frozen-abi can just be symlink
2023-10-21 13:33:10 -05:00
Jeff Washington (jwash) e13756133a
ancient: add many_refs_this_is_newest_alive (#33741)
* add many_refs_this_is_newest_alive

* fix pathological case, add comments

* add log

* update comments

* add log when we fail to pack

* use with_capacity

* fix log comment

* clippy
2023-10-20 09:55:37 -07:00
steviez c98c24bd6d
Revert "Split compute budget instructions process from struct itself … (#33784)
Revert "Split compute budget instructions process from struct itself (#33513)"

This reverts commit c73bebe984. This
was found to be a consensus breaking change.
2023-10-20 15:53:45 +02:00
Brooks ce8ad77373
Uses AccountHash in AppendVec (#33764) 2023-10-19 17:38:09 -04:00
Jeff Washington (jwash) f13c78b7c8
add ancient append vec test (#33762)
* add ancient append vec test

* Update accounts-db/src/ancient_append_vecs.rs

Co-authored-by: Brooks <brooks@prumo.org>

* add some comments

---------

Co-authored-by: Brooks <brooks@prumo.org>
2023-10-19 09:50:38 -07:00
Tao Zhu c73bebe984
Split compute budget instructions process from struct itself (#33513)
* Split compute budget instruction processing from ComputeBudget struct itself, allow compute_budget_instructions be processed elsewhere without having to instantiate ComputeBudget

* updated tests
2023-10-19 11:10:42 -05:00
Brooks 4e5c545e23
Uses AccountHash in tiered storage (#33763) 2023-10-19 11:52:36 -04:00
Brooks 0fcc0a0941
Returns AccountHash from get_filler_account() (#33758) 2023-10-18 20:29:40 -04:00
Brooks 1045548606
Uses AccountHash in StorableAccountsWithHashesAndWriteVersions (#33751) 2023-10-18 18:58:19 -04:00
Brooks 93d882f158
Moves solana-store-tool into the accounts-db crate (#33755) 2023-10-18 16:16:38 -04:00
Brooks 3a580f4df6
StorableAccounts::hash() returns &AccountHash (#33748) 2023-10-18 15:41:26 -04:00
Yueh-Hsuan Chiang 94273434ce
[TieredStorage] HotStorageReader::get_account_meta_from_offset() (#33724)
#### Problem
HotStorageReader currently only implements get_footer().  It does not
have a function to obtain the account meta.

#### Summary of Changes
This PR implements HotStorageReader::get_account_meta_from_offset().
A function that returns the account meta located at the specified offset.
This will be the helper function that will be later used to obtain the account
meta when the offset is available from the index block of a hot storage file. 

#### Test Plan
A new test is included in this PR.
2023-10-18 12:27:39 -07:00
Brooks e96678b302
Uses SeqLock for CachedAccountInner::hash (#33696) 2023-10-18 10:43:35 -04:00
Yueh-Hsuan Chiang 0b05e8db11
[TieredStorage] Footer test for HotStorageReader (#33718)
#### Problem
HotStorageReader currently doesn't have a test that covers its footer.

#### Summary of Changes
This PR includes a test for HotStorageReader that verifies the footer.
2023-10-17 15:24:38 -07:00
Jeff Washington (jwash) c09cbbb778
sort ancient append vec target_slots_sorted (#33729) 2023-10-17 14:49:46 -07:00
Jeff Washington (jwash) 56a7485998
log ancient stats each time we run (#33730) 2023-10-17 14:41:23 -07:00
HaoranYi 673a38c892
move timer after early exit (#33732)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-17 15:20:29 -05:00
HaoranYi f178975185
typo (#33734)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-17 14:45:17 -05:00
Jeff Washington (jwash) 5de9163625
add metric for ancient can't move slots (#33713)
* add metric for ancient can't move slots

* rename

* fix erors in replacing text

* rename
2023-10-17 08:34:18 -07:00
Jeff Washington (jwash) d948e5bf69
ancient shrink on its own cadence (#33712) 2023-10-16 10:06:20 -07:00
HaoranYi 167dac204f
Retry hash file allocation (#33565)
* retry hash file allocation

* add sleep

* submit a datapoint for retry

* typo

* more typos

* Update accounts-db/src/accounts_hash.rs

Co-authored-by: Brooks <brooks@prumo.org>

* fmt

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
Co-authored-by: Brooks <brooks@prumo.org>
2023-10-16 09:21:08 -05:00
Jeff Washington (jwash) c354879a3f
stop padding new append vecs to page size (#33658)
* stop padding new append vecs to page size

* fix test

* fix another test

* for creating test accounts, allocate larger like we used to
2023-10-12 07:11:20 -07:00
HaoranYi 1a2c7f106e
fix typo in comments (#33665)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-12 09:05:10 -05:00
steviez 982d29cf60
Revert "stop padding new append vecs to page size (#33607)" (#33634)
This reverts commit b7962a3610.
2023-10-10 14:43:52 -05:00
Alexander Meißner cb695c7b32
Cleanup - Feature Gate of `enable_early_verification_of_account_modifications` (#33579)
Cleans up the feature gate of enable_early_verification_of_account_modifications:
- Removes PreAccount
- Removes InvokeContext::pre_accounts and InvokeContext::rent
- Removes InvokeContext::verify() and InvokeContext::verify_and_update()
- Removes TransactionContext::is_early_verification_of_account_modifications_enabled()
- Removes TransactionAccounts::is_early_verification_of_account_modifications_enabled
- No longer optional: TransactionContext::rent
2023-10-10 17:26:17 +02:00
Jeff Washington (jwash) b7962a3610
stop padding new append vecs to page size (#33607)
* stop padding new append vecs to page size

* for creating test accounts, allocate larger like we used to
2023-10-10 07:03:37 -07:00
Jeff Washington (jwash) 6c7d3c8aeb
remove redundant page_align (#33608) 2023-10-10 07:02:47 -07:00
Brooks fc73813db2
Adds AccountHash newtype (#33597) 2023-10-09 16:00:52 -04:00
Jeff Washington (jwash) 052677595c
in hash calc, delete old cache files that will not be used earlier (#33432)
* in hash calc, delete old cache files that will not be used earlier

* only delete if supposed to

* fmt
2023-10-09 11:47:39 -07:00
Brooks c924719040
Removes hash param from AccountsCache::store() (#33598) 2023-10-09 18:16:15 +00:00
HaoranYi 72574dac02
Assert acount hash mmap file capacity > 0 (#33575)
assert mmap capacity > 0

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-10-09 09:38:00 -05:00
Brooks bb27bd88d4
Removes write version from tiered storage (#33566) 2023-10-06 16:19:35 -04:00
Yueh-Hsuan Chiang ecb1f8a9d7
[TieredStorage] Include Hot Account in StoredAccountMeta and ReadableAccount (#33544)
#### Problem
All account storage formats are required to implement both StoredAccountMeta
and ReadableAccount, but the implementation for the hot account format is missing.

#### Summary of Changes
This PR includes hot account format into StoredAccountMeta and ReadableAccount
enum.  This will allow the TieredStorageReader in the future PRs to return hot account
format in its `get_account` implementation.
2023-10-06 11:54:02 -07:00
Brooks c66af12bdb
Uses `IntSet` for `RemoveUnrootedSlotsSynchronization::slots_under_contention` (#33530) 2023-10-04 16:34:18 -04:00
Brooks 3b93041401
Uses `IntSet` for dead slots (#33529) 2023-10-04 16:21:44 -04:00
Brooks befc903993
Uses `IntSet` in AccountsDb::calc_delete_dependencies() (#33528) 2023-10-04 16:21:06 -04:00
Brooks f714a44c2a
Uses `IntSet` for `RollingBitField::excess` (#33523) 2023-10-04 15:04:28 -04:00
Brooks 5a9956824f
Uses `IntSet` for `RootsTracker::uncleaned_roots` (#33524) 2023-10-04 13:26:31 -04:00
Brooks d41fa346cc
Uses `IntSet` for `ShrinkCandidates` (#33522) 2023-10-04 13:07:35 -04:00
Brooks daaeb7410e
Adds solana-nohash-hasher (#33521) 2023-10-04 15:56:39 +00:00
HaoranYi bb19ebed49
Convert tuple into dedup result struct (#33450)
convert tupe into dedup result struct

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-29 21:12:50 -05:00
Brooks 0e9e91c65e
Drops the shrink_candidate_slots lock after done inserting (#33459) 2023-09-29 16:58:55 +00:00
Brooks b81ff5d654
Fixup the metrics for remove_dead_accounts_shrink_us (#33458) 2023-09-29 16:04:33 +00:00
HaoranYi 6ea51280dd
Move sort timer out of loop (#33448)
move sort timer out of loop

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-28 20:10:41 -05:00
HaoranYi 25c27d452c
hash dedup vec (#33246)
* hash_dedup vec algo

* reviews

* reviews

* more reviews

* simplify working_set init with add_next_item

* refactor to remove special case "new" from add_item.

The new change is that, even the new item is the new min, it will still be added to working_set.
This change will make init working_set code simpler and the loop loop check simpler.
Since the item is inserted in at the end of the vector, the cost of push into and pop from the working will be O(1), shouldn't affect performance much.

* comments

* refactor unnamed tuple in working set to SlotGroupPointer type

* use SlotGroupPointer in ItemLocation

* Add Copy traits to avoid explicty call of clone on SlotGroupPointer

* consume next in add_next_item fn (credit to jeff).

note that the old code is still correct, since before call to
add_next_item, we will have already overwritten `next` to correct value.

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-28 08:58:08 -05:00
Yihau Chen 3608378097
chore(solana-accounts-db): remove unused deps (#33429) 2023-09-27 16:27:40 +00:00
Yihau Chen ca92e9c387
chore(solana-accounts-db): remove unused deps (#33420) 2023-09-27 11:02:05 +00:00