Commit Graph

145 Commits

Author SHA1 Message Date
Alessandro Decina 16ea141402
sdk: put AccountSharedData::set_data() behind feature="dev-context-only-utils" (#33086)
set_data() is deprecated in favor of set_data_from_slice().
2023-09-01 21:02:14 +07:00
Alexander Meißner 9e703f85de
Upgrades Rust to 1.72.0 & nightly-2023-08-25 (#32961)
* allow pedantic invalid cast lint

* allow lint with false-positive triggered by `test-case` crate

* nightly `fmt` correction

* adapt to rust layout changes

* remove dubious test

* Use transmute instead of pointer cast and de/ref when check_aligned is false.

* Renames clippy::integer_arithmetic to clippy::arithmetic_side_effects.

* bump rust nightly to 2023-08-25

* Upgrades Rust to 1.72.0

---------

Co-authored-by: Trent Nelson <trent@solana.com>
2023-09-01 07:26:13 +00:00
Ryo Onodera 4fe0812f3c
Make should_panic tests compatible with 1.73+ panic format (#33099)
* Adjust should_panic tests for 1.73+ panic format

* Update more panic format change affected tests...
2023-09-01 02:04:22 +00:00
HaoranYi c87f9cdfc9
use shift for pubkey to bin calculation (#33090)
Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-08-31 15:04:25 -05:00
Jeff Washington (jwash) 9a7d503fe4
rework sorting during generate_index (#33076)
* drag data len around during generate_index

* fix tests

* update comments
2023-08-31 12:51:56 -07:00
Brooks 5e26b05597
Removes the "unpacked entries so far" log at startup (#33093) 2023-08-31 15:29:12 -04:00
Ryo Onodera 92a0b310c1
Add #[repr(C)] for more future-proof byte mangling (#33085)
* Add #[repr(C)] for more future-proof byte manglng

* Improve comments after code-review
2023-08-31 22:48:24 +09:00
Jeff Washington (jwash) 1b9c9a313c
reorder generate_index info (#33077)
reorder generateindex info
2023-08-30 16:16:40 -07:00
behzad nouri 4ec5ea6f7b
replaces assert!(matches!(...)) with assert_matches!(...) (#33068)
assert_matches!(...) provides more informative error message when it
fails and it is part of nightly rust:
https://doc.rust-lang.org/std/assert_matches/macro.assert_matches.html
2023-08-30 13:48:27 -04:00
Jeff Washington (jwash) 48f27d5e73
add stats for generate_index (#33061) 2023-08-30 06:43:27 -07:00
Trent Nelson b8dc5daedb
preliminaries for bumping nightly to 2023-08-25 (#33047)
* remove unnecessary hashes around raw string literals

* remove unncessary literal `unwrap()`s

* remove panicking `unwrap()`

* remove unnecessary `unwrap()`

* use `[]` instead of `vec![]` where applicable

* remove (more) unnecessary explicit `into_iter()` calls

* remove redundant pattern matching

* don't cast to same type and constness

* do not `cfg(any(...` a single item

* remove needless pass by `&mut`

* prefer `or_default()` to `or_insert_with(T::default())`

* `filter_map()` better written as `filter()`

* incorrect `PartialOrd` impl on `Ord` type

* replace "slow zero-filled `Vec` initializations"

* remove redundant local bindings

* add required lifetime to associated constant
2023-08-29 23:05:35 +00:00
Jeff Washington (jwash) 00bfc6abd3
remove unused previous_uncleaned_roots (#33033) 2023-08-29 14:54:17 -07:00
Jeff Washington (jwash) 4d452fc5e9
update lru for read only cache if it has been long enough since last access (#32560)
* update lru for read only cache if it has been long enough since last access

* update comment

* use acq/rel
2023-08-29 10:37:55 -07:00
HaoranYi 555741e4d6
Refactor total_items, bin_min/max size calcualtion (#33034)
refactor total_items, bin_min/max size calcuation

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-08-29 10:37:25 -05:00
Jeff Washington (jwash) 9bc09c9610
get unique keys while finding dups (#33039)
* get unique keys while finding dups

* pr feedback
2023-08-29 15:06:35 +00:00
Jeff Washington (jwash) 97d6e6f738
move uncleaned roots code to test mod (#33029) 2023-08-28 14:01:27 -07:00
Jeff Washington (jwash) ce71461bad
remove unused return value from reset_uncleaned_roots (#33030) 2023-08-28 13:17:52 -07:00
Jeff Washington (jwash) 35bd2df0a6
speedup populate_and_retrieve_duplicate_keys_from_startup (#33013) 2023-08-28 10:20:29 -07:00
Jeff Washington (jwash) f4dcb20d2e
add overall measurement for generating index (#33008) 2023-08-28 08:40:54 -07:00
Alexander Meißner a8be70fa7a
General cleanup (#32980)
* Fixes `cargo::` in build.rs

* Fixes overshadowing reexports.

* Removes unnecessary `as *const u8`.

* Removes unnecessary `.into_iter()`.

* cargo clippy
2023-08-24 21:44:19 +02:00
Jon Cinque 0fe902ced7
Bump rand to 0.8, rand_chacha to 0.3, getrandom to 0.2 (#32871)
* sdk: Add concurrent support for rand 0.7 and 0.8

* Update rand, rand_chacha, and getrandom versions

* Run command to replace `gen_range`

Run `git grep -l gen_range | xargs sed -i'' -e 's/gen_range(\(\S*\), /gen_range(\1../'

* sdk: Fix users of older `gen_range`

* Replace `hash::new_rand` with `hash::new_with_thread_rng`

Run:
```
git grep -l hash::new_rand | xargs sed -i'' -e 's/hash::new_rand([^)]*/hash::new_with_thread_rng(/'
```

* perf: Use `Keypair::new()` instead of `generate`

* Use older rand version in zk-token-sdk

* program-runtime: Inline random key generation

* bloom: Fix clippy warnings in tests

* streamer: Scope rng usage correctly

* perf: Fix clippy warning

* accounts-db: Map to char to generate a random string

* Remove `from_secret_key_bytes`, it's just `keypair_from_seed`

* ledger: Generate keypairs by hand

* ed25519-tests: Use new rand

* runtime: Use new rand in all tests

* gossip: Clean up clippy and inline keypair generators

* core: Inline keypair generation for tests

* Push sbf lockfile change

* sdk: Sort dependencies correctly

* Remove `hash::new_with_thread_rng`, use `Hash::new_unique()`

* Use Keypair::new where chacha isn't used

* sdk: Fix build by marking rand 0.7 optional

* Hardcode secret key length, add static assertion

* Unify `getrandom` crate usage to fix linking errors

* bloom: Fix tests that require a random hash

* Remove some dependencies, try to unify others

* Remove unnecessary uses of rand and rand_core

* Update lockfiles

* Add back some dependencies to reduce rebuilds

* Increase max rebuilds from 14 to 15

* frozen-abi: Remove `getrandom`

* Bump rebuilds to 17

* Remove getrandom from zk-token-proof
2023-08-21 19:11:21 +02:00
Jeff Washington (jwash) 524274d8b5
update comment (#32919) 2023-08-21 10:04:37 -07:00
Brooks c6989189ef
Renames CalcAccountsHashFlavor to CalcAccountsHashKind (#32905) 2023-08-21 15:41:07 +00:00
Brooks a563b4cccb
Renames AccountsHashEnum to AccountsHashKind (#32904) 2023-08-21 10:34:56 -04:00
Jeff Washington (jwash) a7d340538e
cache hash files: separate get_file_reference_to_map_later and map (#32891) 2023-08-21 07:30:29 -07:00
Brooks e28c819819
Add docs to load_by_program_slot() (#32896) 2023-08-18 19:11:42 -04:00
Yueh-Hsuan Chiang 66b72a61be
TieredStorage struct (5/N) -- tests for write_accounts (#32850)
#### Problem
Having a PR that is self-contained with a big picture while having it testable
for TieredStorage::write_accounts is challenging, as it requires all the write-side
code, read-side code, and test code.

#### Summary of Changes
This PR solves part of the problem by having one dedicated PR for tests.
Specifically, it includes test utils that allow us to generate test accounts,
invoke write_accounts, and verify the written tiered-storage instance.

With this PR, it will be easier to write future PRs. 

#### Test Plan
A new set of unit-tests and test utils is included in this PR.
2023-08-18 11:29:53 -07:00
Jeff Washington (jwash) 48e51134d7
refactor cache hash data stats (#32884) 2023-08-18 09:24:59 -07:00
Jeff Washington (jwash) 7d203722cb
remove clone() (#32885) 2023-08-18 08:18:32 -07:00
Jeff Washington (jwash) 9499d3d2f8
move `map` to `CacheHashDataFileReference` (#32883)
refactor cache hash data map
2023-08-18 08:12:08 -07:00
Jeff Washington (jwash) 1693af8e68
minor cleanup to read only accounts cache (#32876) 2023-08-18 07:07:40 -07:00
Jeff Washington (jwash) ac6523c373
refactor cache hash data files (#32877) 2023-08-18 07:01:01 -07:00
Jeff Washington (jwash) 9260c0f645
move invariant code outside loop (#32837) 2023-08-17 15:28:04 -07:00
Ryo Onodera e700dde617
dcou: Update qualifier_attr to 0.2.2 (#32838)
Update qualifier_attr to 0.2.2
2023-08-15 13:10:47 +00:00
Brooks e316db2ab4
dcou: set_accounts_hash() (#32829) 2023-08-15 07:18:59 -04:00
steviez 6bbf514e78
Add ability to output components that go into Bank hash (#32632)
When a consensus divergance occurs, the current workflow involves a
handful of manual steps to hone in on the offending slot and
transaction. This process isn't overly difficult to execute; however, it
is tedious and currently involves creating and parsing logs.

This change introduces functionality to output a debug file that
contains the components go into the bank hash. The file can be generated
in two ways:
- Via solana-validator when the node realizes it has diverged
- Via solana-ledger-tool verify by passing a flag

When a divergance occurs now, the steps to debug would be:
- Grab the file from the node that diverged
- Generate a file for the same slot with ledger-tool with a known good
  version
- Diff the files, they are pretty-printed json
2023-08-15 00:12:05 -05:00
Brooks ce57cac370
Use update_accounts_hash() in AccountsHashVerifier (#32830) 2023-08-14 13:11:23 -04:00
HaoranYi bf77a6c7f2
Fix write cache size grow/shrink bug (#32788)
fix write cache size bug

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-08-14 10:11:09 -07:00
Jeff Washington (jwash) 9212ac347c
accounts hash sort_slot_storage_scan sorts in place (#32810) 2023-08-14 07:47:33 -07:00
Ryo Onodera ce63640d6d
dcou: make internal function `pub` conditionally (#32822)
dcou: make internal function pub conditionally
2023-08-13 14:12:20 +09:00
Jeff Washington (jwash) 4e6fb8e343
removed unused accounts hash code (#32791) 2023-08-11 06:49:07 -07:00
HaoranYi d26e3ff22b
Extract read-only cache stats (#32789)
* extract read-only cache stat

* rename stats

---------

Co-authored-by: HaoranYi <haoran.yi@solana.com>
2023-08-10 13:42:09 -05:00
Jeff Washington (jwash) c9f7cb5b7e
keep active stats lock held (#32763) 2023-08-10 08:56:00 -07:00
Brooks 6ff390802b
Uses fold+reduce in de_dup_accounts() (#32765) 2023-08-09 17:27:04 -04:00
Pankaj Garg f4287d70bb
Move accounts-db code to its own crate (#32766) 2023-08-09 13:03:36 -07:00