Commit Graph

3658 Commits

Author SHA1 Message Date
steviez cc8e531a5d
Enforce a minimum of 1 on full and incremental snapshot retention (#30968) 2023-03-30 10:16:36 -05:00
blackghost1987 9f9d6f8a89
Relax `Sized` bound for `Signers` in `Transaction` and `Client` (#30568)
* relax Sized bound for Signers in Transaction and Client

* also relax Sized bounds in client, runtime, thin-client and tpu-client

* add tests for using non-sized transaction signers

* fix macro for thin_client vs ?Sized

* move tests to transactions, add Sized relaxation to mut macro

* fix clippy warning

* get rid of unnecessary imports
2023-03-30 13:12:48 +00:00
Jeff Washington (jwash) d542496d10
fix typo in info log (#30964)
fix typo
2023-03-29 13:23:06 -07:00
Alexander Meißner a6ad37f3ad
Cleanup - `load_and_get_programs_from_cache()` (#30959)
* Replaces assign_program() by replenish() in load_and_get_programs_from_cache().

* Replaces LoadedProgramEntry by a boolean.

* Only lock the global cache once all missing programs have been loaded

* Factors out call of self.rc.accounts.filter_executable_program_accounts().
2023-03-29 16:11:14 +02:00
steviez 4bd7de4887
Update comments around snapshot retention policy unit tests (#30955) 2023-03-29 08:55:05 -05:00
Jeff Washington (jwash) 823d55ea9f
disk index find_entry -> find_index_entry (#30941) 2023-03-29 06:54:03 -07:00
Brooks 1f420f2b03
Moves shrink after flush and clean (#30918) 2023-03-28 16:11:56 -04:00
Pankaj Garg 035c974a8c
Remove redundant check during loading upgradeable program for writing (#30561)
* Remove redundant check during loading upgradeable program

* Apply clippy suggestion
2023-03-28 11:50:49 -07:00
Pankaj Garg aebc191c38
Stop loading program accounts if program exists in cache (#30703)
* Stop loading program accounts if program exists in cache

* load accounts for upgradeable programs

* revert loader change to conditionally use program data account

* load instruction accounts

* generate TransactionExecutorCache from loaded programs

* cleanup account_found_and_dep_index variable

* address review comments

* handle tombstones in loader

* unify tombstone constructor

* handle multiple tombstones
2023-03-28 11:49:56 -07:00
Brooks e7887cfb06
RootBankCache must use Acquire-Release semantics when accessing BankForks::root (#30936) 2023-03-28 14:35:02 -04:00
Jeff Washington (jwash) aaac046b6b
flush individual buckets every n ages (#30855)
disk index flush individual buckets every n ages
2023-03-28 09:26:34 -07:00
Brooks d7ae05c3fd
Unifies logging of start/stop for background services (#30916) 2023-03-28 08:32:18 -04:00
Xiang Zhu bf986d64f8
Add more explict error messages for hardlink operations (#30913) 2023-03-27 19:55:10 -07:00
Pankaj Garg 0df28f6d51
Increment usage count for loaded programs and call eviction (#30900) 2023-03-27 06:48:05 -07:00
Andrew Fitzgerald 2a05b25732
Standardize Thread Naming - Prioritization Fee Cache (#30890)
Thread naming in accordance with #27237
2023-03-24 13:48:55 -07:00
Andrew Fitzgerald f226a34f48
Only need bank reference for update (#30879) 2023-03-24 09:48:04 -07:00
Tao Zhu 3e500d9e92
Feature gate builtin consumes static units during processing instruction (#30702)
* add feature gate
* builtins consume statically defined units at beginning of process_instruction()
* Add new instructionError; return error if builtin did not consume units to enforce builtin to consume units;
* updated related tests
* updated ProgramTest with deactivated native_programs_consume_cu feature to continue support existing mock/test programs that do not consume units
2023-03-24 11:31:01 -05:00
Jeff Washington (jwash) 3ebfda5aef
add stat for disk buckets flushed (#30850) 2023-03-22 15:04:17 -07:00
Xiang Zhu d69f60229d
Construct bank from snapshot dir (#30171)
* Construct a bank from a snapshot directory

* Clean up.  Remove archiving releated data structures

* Fix auto checks

* remove ArchiveFormat::None

* fix merge error

* remove incremental snapshot dir

* Minor cleanup, remove unused functiond defs

* remove pub fn bank_from_latest_snapshot_dir

* rename bank_from_snapshot_dir to bank_from_snapshot

* Clean up invalid comments

* A few minor review changes

* Removed insert_slot_storage_file

* Add comment explain hardlink symlink

* Skip the whole verify_snapshot_bank call for the from_dir case

* Add bank.set_initial_accounts_hash_verification_completed()

* address review issues: appendvec to append_vec, replace unwrap with expect, etc

* AtomicAppendVecId, remove arc on bank etc

* slice, CI error on &snapshot_version_path

* measure_build_storage

* move snapshot_from

* remove measure_name from build_storage_from_snapshot_dir

* remove from_dir specific next_append_vec_id logic

* revert insert_slot_storage_file change

* init next_append_vec_id to fix the substraction underflow

* remove measure from build_storage_from_snapshot_dir

* make measure name more specific

* refactor status_cache deserialization into a function

* remove reference to pass the ci check

* track next appendvec id

* verify that the next_append_vec_id tracking is correct

* clean up usize

* in build_storage_from_snapshot_dir remove expect

* test max appendvecc id tracking with multiple banks in the test

* cleared expect and unwrap in streaming_snapshot_dir_files

* rebase cleanup

* change to measure!

* dereference arc in the right way
2023-03-22 13:49:23 -07:00
Jeff Washington (jwash) 9a1d5ea95d
at startup, keep duplicates in in-memory index since they will be cleaned shortly (#30736)
at startup, keep duplicates in in-memory index since they will be cleaned soon
2023-03-22 08:33:39 -07:00
Brooks 35437b8dad
Makes AccountsHashVerifier aware of Incremental Accounts Hash (#30820) 2023-03-22 10:20:16 -04:00
Tao Zhu 21c287a64f
include loaded accounts data size limit in transaction fee calculation (#30659)
* include loaded accounts data size limit in transaction base fee calculation
* citing compute_budget for heap cost;
* update sbf tests

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2023-03-21 18:41:41 -05:00
HaoranYi d646820a6e
Do not log detailed epoch stakes (#30825)
* do not log detailed epoch stakes

* allow dead code

* log stake details at trace level

* no need dead_code

* Update runtime/src/bank.rs

Co-authored-by: Brooks <brooks@prumo.org>

---------

Co-authored-by: Brooks <brooks@prumo.org>
2023-03-21 14:43:06 -05:00
Jeff Washington (jwash) 2216647f7e
better duplicate key stats during index generation (#30829) 2023-03-21 12:02:23 -07:00
Brooks b64d0de771
Makes snapshot_utils aware of Incremental Accounts Hash (#30804) 2023-03-21 16:34:30 +00:00
HaoranYi 6a6d5ee498
typo (#30824) 2023-03-21 11:19:16 -05:00
Brooks 836494fc5f
Cleans *up to* the last full snapshot slot in verify_snapshot_bank() (#30818)
Cleans up to the last full snapshot slot in verify_snapshot_bank()
2023-03-21 15:49:49 +00:00
Brooks e5edcb9c48
Sets IncrementalAccountsHash in serde_snapshot when reconstructing AccountsDb (#30801) 2023-03-21 10:51:04 -04:00
Jeff Washington (jwash) ce0e23fbab
accounts index does not randomly flush dirty entries (#30783) 2023-03-20 12:23:30 -07:00
Yueh-Hsuan Chiang 72c6099ea0
Move alignment related consts from append_vec.rs to accounts_file.rs (#30782)
#### Problem
As we start supporting new storage formats, alignment-related constants and macros
defined in append_vec.rs aren't only specific to AppendVec.

#### Summary of Changes
Move alignment-related constants/macros from append_vec.rs to accounts_file.rs
2023-03-20 11:34:18 -07:00
HaoranYi f42f61804f
typos (#30797) 2023-03-20 10:47:15 -05:00
Jeff Washington (jwash) df2c95119c
add buckets stats for # files and total file size (#30780)
* add buckets stats for # files and total file size

* change metrics to bytes

* get index file size correct
2023-03-20 08:22:31 -07:00
Brooks 989c80342b
Adds plumbing and tests for verify_snapshot_bank() with Incremental Accounts Hash (#30789) 2023-03-19 21:44:38 -04:00
Brooks d3273ba118
Removes unused `ancestors` param from `get_snapshot_storages` (#30773) 2023-03-18 17:17:41 -04:00
Brooks 8325b0f591
Do not use ancestors in get_snapshot_storages (#30772)
* Do not use ancestors in get_snapshot_storages

* fixup verify_accounts_hash bench
2023-03-17 22:48:47 +00:00
Xiang Zhu 8e3a30c22c
Clean orphaned account snapshot dirs (#30645)
* Clean up orphaned account snapshot hardlink dirs

* fix compilation issues

* debugged, now working.  seeing the orphaned directories deleted

* change back to eprintln + exit for account_path error

* changed eprintln to panic for now

* add test_clean_orphaned_account_snapshot_dirs for codecov check

* address a few comments and nit isseus

* directly unzip, skipped the intermediate array of tuples

* let set_up_account_run_and_snapshot_paths return Result

* 'proper' typo, and comment on return

* use map_err

* use for loop in clean_orphaned_account_snapshot_dirs, removed panic

* add test_set_up_account_run_and_snapshot_paths

* minor, replace .for_each with .all

* rename set_up_account_run_and_snapshot_paths to create_all_accounts_run_and_snapshot_dirs

* remove unnecessary closure return type

* change to for loop

* change match to unwrap_or_else

* remove create_dir_all(&account_path) in create_all

* minor comment cleanup
2023-03-17 15:22:10 -07:00
Brooks d4dd4360e7
Removes `base_slot` param from `calculate_incremental_accounts_hash()` (#30774) 2023-03-17 15:22:22 -04:00
Ryo Onodera 05ee06814e
Fix wrongly swapped parent slot and block height (#30744) 2023-03-17 20:07:52 +09:00
Tao Zhu 17ba051e1f
Revert "add default_cost as mandatory field for Builtin (#30639)" (#30749)
This reverts commit 7b95c8e698.
2023-03-16 11:21:06 -05:00
Brooks 8be99c1c7d
Verifies incremental accounts hash in verify_accounts_hash_and_lamports (#30735) 2023-03-16 09:48:06 -04:00
Yueh-Hsuan Chiang fce949009d
Add new_from_file() API to AccountsFile (#30687)
#### Problem
Accounts db currently use AppendVec::new_from_file() directly
to create a new AcountsFile instance from an existing file.
However, this method should be abstracted out to AccountsFile
so that an existing file can be opened correctly using the right format.

#### Summary of Changes
Add new_from_file() API to AccountsFile which will open an existing
file based on its accounts file format.

Currently, it only supports AppendVec.
2023-03-15 20:38:20 -07:00
Brooks cbc0d01d0d
Refactors verify_accounts_hash_and_lamports() (#30733) 2023-03-15 18:37:27 -04:00
Brooks cefb00e3fb
Includes error in panic in initial accounts hash verification (#30725) 2023-03-15 12:24:55 -04:00
Brooks 65cd55261a
Makes Bank::get_snapshot_hash() aware of incremental accounts hashes (#30719) 2023-03-14 18:03:38 -04:00
Brooks 93c43610ac
AccountsHashVerifier stores IncrementalAccountsHash in AccountsDb (#30696) 2023-03-14 12:41:44 -04:00
Jeff Washington (jwash) 89d5efa0d8
mem idx tracks duplicates better (#30701) 2023-03-14 07:26:39 -07:00
Brooks 560ec08d5e
AccountsHashVerifier writes BankIncrementalSnapshotPersistence (#30587) 2023-03-13 17:44:34 -04:00
Brooks 263342a686
Adds accounts hashes from full & incremental snapshots at startup (#30656) 2023-03-13 14:47:25 -04:00
Brooks 505e3ff5c7
AccountsHashVerifier updates AccountsDb after calculating accounts hash (#30658) 2023-03-13 16:41:24 +00:00
Madhav Goyal 9a8a0223ca
Link needs updating (#30688)
* Link needs updating

Link to monitor the solana cluster doesnt work

* Updated to solana grafna

users could fin the dashboard they are looking for
2023-03-13 17:33:22 +01:00