Commit Graph

74 Commits

Author SHA1 Message Date
Jeff Washington (jwash) 3ad8394047
stop removing empty bins during index generation (#33242) 2023-09-14 07:53:33 -07:00
Jeff Washington (jwash) 886eabd74d
gen idx: refactor StorageSizeAndCount population (#33244) 2023-09-14 07:53:05 -07:00
Jeff Washington (jwash) 602f0f6ecb
generate index: fn arg: &Slot -> Slot (#33229)
fn arg: &Slot -> Slot
2023-09-13 14:14:30 -07:00
Jeff Washington (jwash) c2bf2a969d
remove unused metrics (#33241) 2023-09-13 14:14:10 -07:00
Jeff Washington (jwash) cb310a310a
set approx store count at generate index time (#33240) 2023-09-13 14:13:46 -07:00
Jeff Washington (jwash) 093ae61e85
single pass through each storage to populate info (#33227) 2023-09-13 12:37:01 -07:00
Jeff Washington (jwash) 3ffd78f2d7
add comments (#33226) 2023-09-12 13:51:27 -07:00
Brooks 21e7f98b3a
Uses hex for the hash in accounts hash cache file names (#33196) 2023-09-12 11:26:08 -04:00
Brooks c61ee20b44
Transient accounts hash cache dir is unnecessary (#33181) 2023-09-12 11:23:07 -04:00
HaoranYi 3cd3994939
clean up pub crate (#33214)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-11 19:28:26 -05:00
Brooks 09936aac0e
Removes `new()` from POD CalculateHashIntermediate and CumulativeOffset (#33211) 2023-09-11 22:33:25 +00:00
HaoranYi cf35799b2a
remove unused stats (#33213)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-11 21:03:37 +00:00
Brooks 659fc6837b
Removes `pub` from accounts_hash.rs items (#33209) 2023-09-11 20:49:51 +00:00
Jeff Washington (jwash) d724af863c
add a few accounts hash dedup tests (#33208) 2023-09-11 13:20:28 -07:00
Brooks 6298c6c31e
Shares accounts hash cache data between full and incremental (#33164) 2023-09-11 15:55:48 -04:00
Alexander Meißner bbb57be0a5
Refactor - Move interfaces of address-lookup-table into the program SDK (#33165)
* Adds a module `address_lookup_table` to the SDK.

* Adds a module `address_lookup_table::instruction` to the SDK.

* Adds a module `address_lookup_table::error` to the SDK.

* Adds a module `address_lookup_table::state` to the SDK.

* Moves AddressLookupTable into SDK as well.

* Moves AddressLookupTableAccount into address_lookup_table.

* Adds deprecation messages.

* Disentangles dependencies across cargo files.
2023-09-11 21:10:40 +02:00
Jeff Washington (jwash) 4dfe62a2f0
rework accounts hash calc dedup to avoid kernel file error (#33195)
* in hash calc, calculate max_inclusive_num_pubkeys

* in hash calc, dedup uses mmap files to avoid os panic

* as_mut_ptr

* remove unsafe code

* refactor count in hash files

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-11 10:57:21 -07:00
Jeff Washington (jwash) a145ade564
stat for time spent copying generate index contents (#33187)
* stat for time spent copying generate index contents

* rework to move stat to generate_index

* fix fmt
2023-09-08 12:55:54 -07:00
Jeff Washington (jwash) dc6b1eb653
in shrink, get stores prior to shrink starting (#33194) 2023-09-08 08:45:32 -07:00
Jeff Washington (jwash) 0083e426b2
ShrinkCandidates only holds slot (#33173) 2023-09-07 12:28:05 -07:00
Jeff Washington (jwash) e331275178
remove coupling of ShrinkCandidates to HashMap (#33176) 2023-09-07 09:07:37 -07:00
Brooks f4816dc0cf
Moves CacheHashData test-only methods into tests module (#33170) 2023-09-06 15:25:58 -04:00
Brooks a3dc3eb37c
Takes inner HashSet when dropping CacheHashData (#33169) 2023-09-06 18:19:01 +00:00
behzad nouri 88ee8f5820
replaces once_cell::sync::OnceCell with std::sync::OnceLock (#33140)
std::sync::OnceLock has become stable since rust 1.70.0 and there is no
longer a need for an external crate dependency.
2023-09-06 16:46:51 +00:00
HaoranYi a8bc6ebe38
Use `write_all` for binned account hash file writes (#33095)
write_all

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-09-06 11:03:05 -05:00
Illia Bobyr d077b13efa
accounts-db: test_hash_stored_account: Avoid UB. (#33083)
unsafe { transmute }` in the test is generating undefined behavior, as it
assigns a value into `bool` that is neither 0 nor 1.

We see discrepancy between release and debug builds due to this.

It is better to avoid `unsafe` blocks if there alternatives that let the
compiler check everything.
2023-09-05 20:30:17 -07:00
Jeff Washington (jwash) fb1ba216f5
Revert "Add an RPC API that can query the list of Top N secondary index keys and their sizes (#28887)" (#33121)
* Revert "Add an RPC API that can query the list of Top N secondary index keys and their sizes (#28887)"

This reverts commit 1e3d6349aa.

* Revert "Add Admin RPC Front End for Top N Secondary Index Key Sizes Query. (#29352)"

This reverts commit aa353e4b83.

* fix test uses

* fmt
2023-09-01 14:36:06 -07:00
Brooks d1b849972f
Support a custom path for the accounts hash cache (#33115) 2023-09-01 15:29:56 -04:00
Jeff Washington (jwash) 255029f61a
add rate info to generate index log (#33108) 2023-09-01 11:40:00 -07:00
Alessandro Decina 16ea141402
sdk: put AccountSharedData::set_data() behind feature="dev-context-only-utils" (#33086)
set_data() is deprecated in favor of set_data_from_slice().
2023-09-01 21:02:14 +07:00
Alexander Meißner 9e703f85de
Upgrades Rust to 1.72.0 & nightly-2023-08-25 (#32961)
* allow pedantic invalid cast lint

* allow lint with false-positive triggered by `test-case` crate

* nightly `fmt` correction

* adapt to rust layout changes

* remove dubious test

* Use transmute instead of pointer cast and de/ref when check_aligned is false.

* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.

* bump rust nightly to 2023-08-25

* Upgrades Rust to 1.72.0

---------

Co-authored-by: Trent Nelson <trent@solana.com>
2023-09-01 07:26:13 +00:00
Ryo Onodera 4fe0812f3c
Make should_panic tests compatible with 1.73+ panic format (#33099)
* Adjust should_panic tests for 1.73+ panic format

* Update more panic format change affected tests...
2023-09-01 02:04:22 +00:00
HaoranYi c87f9cdfc9
use shift for pubkey to bin calculation (#33090)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-08-31 15:04:25 -05:00
Jeff Washington (jwash) 9a7d503fe4
rework sorting during generate_index (#33076)
* drag data len around during generate_index

* fix tests

* update comments
2023-08-31 12:51:56 -07:00
Brooks 5e26b05597
Removes the "unpacked entries so far" log at startup (#33093) 2023-08-31 15:29:12 -04:00
Ryo Onodera 92a0b310c1
Add #[repr(C)] for more future-proof byte mangling (#33085)
* Add #[repr(C)] for more future-proof byte manglng

* Improve comments after code-review
2023-08-31 22:48:24 +09:00
Jeff Washington (jwash) 1b9c9a313c
reorder generate_index info (#33077)
reorder generateindex info
2023-08-30 16:16:40 -07:00
behzad nouri 4ec5ea6f7b
replaces assert!(matches!(...)) with assert_matches!(...) (#33068)
assert_matches!(...) provides more informative error message when it
fails and it is part of nightly rust:
https://doc.rust-lang.org/std/assert_matches/macro.assert_matches.html
2023-08-30 13:48:27 -04:00
Jeff Washington (jwash) 48f27d5e73
add stats for generate_index (#33061) 2023-08-30 06:43:27 -07:00
Trent Nelson b8dc5daedb
preliminaries for bumping nightly to 2023-08-25 (#33047)
* remove unnecessary hashes around raw string literals

* remove unncessary literal `unwrap()`s

* remove panicking `unwrap()`

* remove unnecessary `unwrap()`

* use `[]` instead of `vec![]` where applicable

* remove (more) unnecessary explicit `into_iter()` calls

* remove redundant pattern matching

* don't cast to same type and constness

* do not `cfg(any(...` a single item

* remove needless pass by `&mut`

* prefer `or_default()` to `or_insert_with(T::default())`

* `filter_map()` better written as `filter()`

* incorrect `PartialOrd` impl on `Ord` type

* replace "slow zero-filled `Vec` initializations"

* remove redundant local bindings

* add required lifetime to associated constant
2023-08-29 23:05:35 +00:00
Jeff Washington (jwash) 00bfc6abd3
remove unused previous_uncleaned_roots (#33033) 2023-08-29 14:54:17 -07:00
Jeff Washington (jwash) 4d452fc5e9
update lru for read only cache if it has been long enough since last access (#32560)
* update lru for read only cache if it has been long enough since last access

* update comment

* use acq/rel
2023-08-29 10:37:55 -07:00
HaoranYi 555741e4d6
Refactor total_items, bin_min/max size calcualtion (#33034)
refactor total_items, bin_min/max size calcuation

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-08-29 10:37:25 -05:00
Jeff Washington (jwash) 9bc09c9610
get unique keys while finding dups (#33039)
* get unique keys while finding dups

* pr feedback
2023-08-29 15:06:35 +00:00
Jeff Washington (jwash) 97d6e6f738
move uncleaned roots code to test mod (#33029) 2023-08-28 14:01:27 -07:00
Jeff Washington (jwash) ce71461bad
remove unused return value from reset_uncleaned_roots (#33030) 2023-08-28 13:17:52 -07:00
Jeff Washington (jwash) 35bd2df0a6
speedup populate_and_retrieve_duplicate_keys_from_startup (#33013) 2023-08-28 10:20:29 -07:00
Jeff Washington (jwash) f4dcb20d2e
add overall measurement for generating index (#33008) 2023-08-28 08:40:54 -07:00
Alexander Meißner a8be70fa7a
General cleanup (#32980)
* Fixes `cargo::` in build.rs

* Fixes overshadowing reexports.

* Removes unnecessary `as *const u8`.

* Removes unnecessary `.into_iter()`.

* cargo clippy
2023-08-24 21:44:19 +02:00
Jon Cinque 0fe902ced7
Bump rand to 0.8, rand_chacha to 0.3, getrandom to 0.2 (#32871)
* sdk: Add concurrent support for rand 0.7 and 0.8

* Update rand, rand_chacha, and getrandom versions

* Run command to replace `gen_range`

Run `git grep -l gen_range | xargs sed -i'' -e 's/gen_range(\(\S*\), /gen_range(\1../'

* sdk: Fix users of older `gen_range`

* Replace `hash::new_rand` with `hash::new_with_thread_rng`

Run:
```
git grep -l hash::new_rand | xargs sed -i'' -e 's/hash::new_rand([^)]*/hash::new_with_thread_rng(/'
```

* perf: Use `Keypair::new()` instead of `generate`

* Use older rand version in zk-token-sdk

* program-runtime: Inline random key generation

* bloom: Fix clippy warnings in tests

* streamer: Scope rng usage correctly

* perf: Fix clippy warning

* accounts-db: Map to char to generate a random string

* Remove `from_secret_key_bytes`, it's just `keypair_from_seed`

* ledger: Generate keypairs by hand

* ed25519-tests: Use new rand

* runtime: Use new rand in all tests

* gossip: Clean up clippy and inline keypair generators

* core: Inline keypair generation for tests

* Push sbf lockfile change

* sdk: Sort dependencies correctly

* Remove `hash::new_with_thread_rng`, use `Hash::new_unique()`

* Use Keypair::new where chacha isn't used

* sdk: Fix build by marking rand 0.7 optional

* Hardcode secret key length, add static assertion

* Unify `getrandom` crate usage to fix linking errors

* bloom: Fix tests that require a random hash

* Remove some dependencies, try to unify others

* Remove unnecessary uses of rand and rand_core

* Update lockfiles

* Add back some dependencies to reduce rebuilds

* Increase max rebuilds from 14 to 15

* frozen-abi: Remove `getrandom`

* Bump rebuilds to 17

* Remove getrandom from zk-token-proof
2023-08-21 19:11:21 +02:00