Commit Graph

3189 Commits

Author SHA1 Message Date
dependabot[bot] a00c045d7c
chore: bump once_cell from 1.12.0 to 1.13.0 (#27049)
* chore: bump once_cell from 1.12.0 to 1.13.0

Bumps [once_cell](https://github.com/matklad/once_cell) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.12.0...v1.13.0)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2022-08-10 12:19:10 -06:00
dependabot[bot] ae5b680c6f
chore: bump serde from 1.0.138 to 1.0.143 (#27015)
* chore: bump serde from 1.0.138 to 1.0.143

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.138 to 1.0.143.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.138...v1.0.143)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-08-10 05:08:43 +00:00
Michael Vines ccfbc54195 Move vote program state and instructions to solana-program 2022-08-09 20:52:47 -07:00
apfitzge 6e57a8f8a1
A more convenient store-tool (#26796)
* Use new_from_file_unchecked - don't sanitize input length for appendvec file

* Exit-early on completely zeroed accounts
2022-08-08 06:55:00 -07:00
Will Hickey ed8c224374
Bump version to v1.12 (#26967) 2022-08-06 13:20:30 -05:00
Christian Kamm cf58640937
Keypair: implement clone() (#26248)
* Keypair: implement clone()

This was not implemented upstream in ed25519-dalek to force everyone to
think twice before creating another copy of a potentially sensitive
private key in memory.

See https://github.com/dalek-cryptography/ed25519-dalek/issues/76

However, there are now 9 instances of
  Keypair::from_bytes(&keypair.to_bytes())
in the solana codebase and it would be preferable to have a function.

In particular since this also comes up when writing programs and can
cause users to either start messing with lifetimes or discover the
from_bytes() workaround themselves.

This patch opts to not implement the Clone trait. This avoids automatic
use in order to preserve some of the original "let developers think
twice about this" intention.

* Use Keypair::clone
2022-08-06 11:54:38 -06:00
Justin Starry 69598ed4c0
Refactor: Add `RuntimeConfig` field to Bank (#26946)
* Refactor: Simplify arguments for bank constructor methods

* Refactor: Add RuntimeConfig to Bank fields

* Arc wrap runtime_config

* Arc wrap all runtime config usages

* Remove Copy trait derivation from RuntimeConfig

* Remove some arc wrapping
2022-08-05 20:49:00 +01:00
Brooks Prumo 06a610130e
Remove bank test_max_accounts_data_size_exceeded() (#26772)
bank: remove test_max_accounts_data_size_exceeded()
2022-08-05 09:55:29 -04:00
Jeff Washington (jwash) 73fe400407
protect access to rent paying account vec (#26919) 2022-08-05 08:19:46 -05:00
Tyera Eulberg 2dca239480
Remove runtime dependency from solana-transaction-status (#26930)
* Move RewardType out of runtime

* Move collect_token_balances to solana-ledger

* Remove solana-runtime dependency
2022-08-05 00:20:27 -06:00
steviez 300666dce7
Make `solana-ledger-tool` run AccountsBackgroundService (#26914)
Prior to this change, long running commands like `solana-ledger-tool
verify` would OOM due to AccountsDb cleanup not happening.

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-04 15:44:31 -05:00
Jeff Washington (jwash) 4b8f881af3
in mem acct idx scan multiple ages simultaneously (#26853) 2022-08-03 13:44:01 -05:00
github-actions[bot] fbf1bf6d86
Bump Version to 1.11.6 (#26906)
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
2022-08-03 12:48:43 -05:00
Jeff Washington (jwash) 0732a6c45f
add 'oldest_root' stat in hash calc (#26896) 2022-08-03 10:02:08 -05:00
Jeff Washington (jwash) 6706be0f56
log oldest dirty slot on clean (#26897) 2022-08-03 10:01:29 -05:00
Brennan Watt f24de9d254
Make tx error accounting more granular (#26718) 2022-08-02 16:43:03 -07:00
Jeff Washington (jwash) 224550d65f
allow caching on ancient append vecs for accounts hash calc (#26885) 2022-08-02 14:40:29 -05:00
Jeff Washington (jwash) acc1112e9c
calculate cache hash path for store_detailed_debug_info_on_failure (#26874) 2022-08-02 11:19:00 -05:00
Jeff Washington (jwash) e5c5055869
Revert "allow caching on ancient append vecs for accounts hash calc" (#26882)
Revert "allow caching on ancient append vecs for accounts hash calc (#26770)"

This reverts commit 30b408ed0c.
2022-08-02 10:23:11 -05:00
Jon Cinque f210182851
accounts: Don't collect rent on newly created accounts (#26851) 2022-08-02 16:34:34 +02:00
Jeff Washington (jwash) 050c66f197
enable accounts hash caching if store_detailed_debug_info_on_failure (#26864) 2022-08-01 14:27:20 -05:00
Jeff Washington (jwash) d159682a6d
add comment (#26875) 2022-08-01 14:25:19 -05:00
Jeff Washington (jwash) ebabc53cee
use Vec::drain instead of option/take (#26852) 2022-07-31 21:04:15 -05:00
Jeff Washington (jwash) c7462b7a52
ledger tool verify can store debug info on hash calc (#26837) 2022-07-29 15:54:56 -05:00
HaoranYi 38cd29810f
Fix ordering for AccountEntry ref_count (#26842)
fix ordering for ref_count
2022-07-29 15:47:03 -05:00
Jeff Washington (jwash) 69b5dd9b07
cleanup ignore_eof_error (#26838) 2022-07-29 12:10:22 -05:00
apfitzge c1f7d1a367
Add AppendVec::new_from_file_unchecked (#26795)
new_from_file_unchecked
2022-07-28 14:57:26 -05:00
Brennan Watt 467cb5def5
Concurrent slot replay (#26465)
* Concurrent replay slots

* Split out concurrent and single bank replay paths

* Sub function processing of replay results for readability

* Add feature switch for concurrent replay
2022-07-28 11:33:19 -07:00
Jeff Washington (jwash) 817f65bb50
add full_snapshot to hash config (#26811) 2022-07-28 09:46:34 -05:00
Ashwin Sekar 8d69e8d447
Compact vote state updates to reduce block size (#26616)
* Compact vote state updates to reduce block size

* Add rpc transaction tests
2022-07-27 13:23:44 -06:00
Jeff Washington (jwash) 1a5b830294
CalcAccountsHashConfig default for tests (#26810) 2022-07-27 13:21:38 -05:00
Jeff Washington (jwash) 1421833c5e
log when shrink still contains extra stores (#26794) 2022-07-27 09:14:16 -05:00
Jeff Washington (jwash) 30b408ed0c
allow caching on ancient append vecs for accounts hash calc (#26770) 2022-07-27 09:13:32 -05:00
dependabot[bot] cd4340ed06
chore: bump bytemuck from 1.10.0 to 1.11.0 (#26792)
* chore: bump bytemuck from 1.10.0 to 1.11.0

Bumps [bytemuck](https://github.com/Lokathor/bytemuck) from 1.10.0 to 1.11.0.
- [Release notes](https://github.com/Lokathor/bytemuck/releases)
- [Changelog](https://github.com/Lokathor/bytemuck/blob/main/changelog.md)
- [Commits](https://github.com/Lokathor/bytemuck/compare/v1.10.0...v1.11.0)

---
updated-dependencies:
- dependency-name: bytemuck
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-07-26 20:50:30 -06:00
apfitzge 293d651ca8
store-tool use accounts_iter (#26793) 2022-07-26 17:29:53 -05:00
Tao Zhu 5e622f82ac
Add feature gate to use default units per instruction for fee calculation (#26786)
* Add feature gate to use default units per instruction for fee calculation

* fix bpf tests
2022-07-26 19:39:24 +00:00
apfitzge 5de1a7accf
Add streaming_unpack_snapshot to send entry files during unpack (#26781)
* Add streaming_unpack_snapshot to send entry files during unpack

* Update unpack tests to call internal function
2022-07-26 14:09:28 -05:00
apfitzge a7e4df5d6e
Separate remapping append_vecs into functions (#26696)
Separate remapping append_vecs into remap_and_reconstruct_storages and helpers
2022-07-25 14:55:30 -05:00
Jeff Washington (jwash) 6f5995c841
add stats for disk_index_find_entry_mut_us (#26753) 2022-07-25 14:47:17 -05:00
github-actions[bot] 5d038b9d2a
Bump Version to 1.11.5 (#26758)
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
2022-07-25 13:05:14 -06:00
Brooks Prumo 01e367f19b
Add Bank::update_accounts_data_size_delta_off_chain_for_tests() (#26771) 2022-07-25 18:58:04 +00:00
Jeff Washington (jwash) d9c7bc7e78
Revert "cleanup feature: default units per instruction (#26684)" (#26750)
This reverts commit 39a34db52a.
2022-07-23 11:03:46 -05:00
apfitzge f2b81fe665
Separate field_from_stream to get bank/account fields from individual snapshot stream (#26679) 2022-07-21 12:27:31 -05:00
Alexander Meißner 075a5ac44e
Removes `AccountSharedData` from `SysvarCache`. (#26712)
* Removes AccountSharedData from SysvarCache.

* Fixes incorrect transaction account index in stake_instruction::test_set_lockup().
2022-07-21 18:32:28 +02:00
Alexander Meißner cbb74a190f
Cleanup: `record_instruction_in_transaction_context_push` (#26658)
Cleanup feature gate of record_instruction_in_transaction_context_push.
2022-07-21 12:49:34 +02:00
HaoranYi 6ec89ee861
log error for cache_hash_data save failure (#26676)
log error for cach_hash_data save
2022-07-20 16:19:35 -05:00
Jack May 39a34db52a
cleanup feature: default units per instruction (#26684) 2022-07-20 19:13:34 +00:00
apfitzge b20edd232b
Remove useless enumerate from test_hold_range_in_memory2 (#26683) 2022-07-20 10:06:08 -05:00
Jeff Washington (jwash) 9f2f2a45cd
AcctIdx: ref_count_from_storage remove Arc::clone (#26678) 2022-07-20 10:05:13 -05:00
Justin Starry f8e9af5f1e
Cap the number of accounts passed to a bpf program ix (#26630)
* Cap the number of accounts passed to a bpf program ix

* update bank abi hash

* fix ci failures
2022-07-20 14:12:43 +02:00
apfitzge 07f41628db
untar_snapshot_create_shared_buffer doesn't need to return a result (#26686) 2022-07-19 23:15:41 +00:00
Jeff Washington (jwash) 2d689ac53a
get rid of rwlock per bin in accounts index (#26675) 2022-07-19 17:48:40 -05:00
apfitzge e7cd6daebe
Add helper fn to create shared buffer - simplify archive format matching (#26674) 2022-07-19 20:30:30 +00:00
apfitzge 84c8cfe901
Refactor/hardened unpack more generic unpack archive (#26677)
* Add entry_processor fn to unpack_archive - not used yet

* Add generic unpack_snapshot_with_processors to separate account paths handling from entry checking
2022-07-19 14:48:57 -05:00
Jeff Washington (jwash) 7efe72a74f
eliminate write lock usage in acct idx (#26673) 2022-07-19 13:24:03 -05:00
apfitzge 77843306c7
rename parallel_archivers -> parallel_divisions for consistency (#26672)
parallel_archivers -> parallel_divisions for consistency
2022-07-19 13:15:33 -05:00
Jeff Washington (jwash) 263911e7fd
save off what we find when calculating hash (#26663) 2022-07-19 09:55:52 -05:00
Jeff Washington (jwash) 738c4a79b7
metric for 'ancient_account_cleans' (#26664) 2022-07-19 09:16:23 -05:00
Jeff Washington (jwash) 6efe7e095b
temporarily make ancient append vecs 10x smaller to find intermittent bugs (#26574) 2022-07-19 08:51:36 -05:00
Jeff Washington (jwash) c4d2c95540
move hash stat to correct level for ahv to work (#26649) 2022-07-18 12:44:33 -05:00
Jeff Washington (jwash) 1c08f83c7e
eliminate flatten().collect() of reclaims in clean (#26647) 2022-07-18 12:37:17 -05:00
Justin Starry fd6038884b Clean up forgotten error log 2022-07-17 12:21:52 +02:00
Jeff Washington (jwash) be8813e588
shrink stops using HandleReclaims (#26594) 2022-07-15 17:54:48 -05:00
Jeff Washington (jwash) 83c614da3c
change type to be more clear (#26646) 2022-07-15 17:17:17 -05:00
Jeff Washington (jwash) 47716a5e01
async hash verify on load (#26208)
* verify accounts hash in bg on startup

* fix some tests and loading from genesis

* add extra state for when background thread has completed
2022-07-15 14:29:56 -05:00
behzad nouri bf225bae73
restricts rent-paying accounts lifetime extension (#26606)
https://github.com/solana-labs/solana/pull/22292
prevents rent paying accounts creation going forward. However a rent
paying account can linger on for ever if it is continually topped up but
stays below the rent-exempt minimum.
This can prevent eliminating accounts-rewrites and the problematic
rent_epoch field in accounts.

Link to discord discussion:
https://discord.com/channels/428295358100013066/943609352068145162/995202300001927219

This commit restricts rent-paying accounts lifetime extension by
preventing increasing lamports on the account if the account stays below
the rent-exempt minimum.
2022-07-15 13:23:37 +00:00
Alexander Meißner 038da82b6f
Feature: Early verification of account modifications in `BorrowedAccount` (#25899)
* Adjusts test cases for stricter requirements.

* Removes account reset in deserialization test.

* Removes verify related test cases.

* Replicates account modification verification logic of PreAccount in BorrowedAccount.

* Adds TransactionContext::account_touched_flags.

* Adds account modification verification to the BPF ABIv0 and ABIv1 deserialization, CPI syscall and program-test.

* Replicates the total sum of all lamports verification of PreAccounts in InstructionContext

* Check that the callers instruction balance is maintained during a call / push.

* Replicates PreAccount statistics in TransactionContext.

* Disable verify() and verify_and_update() if the feature enable_early_verification_of_account_modifications is enabled.

* Moves Option<Rent> of enable_early_verification_of_account_modifications into TransactionContext::new().

* Relaxes AccountDataMeter related test cases.

* Don't touch the account if nothing changes.

* Adds two tests to trigger InstructionError::UnbalancedInstruction.

Co-authored-by: Justin Starry <justin@solana.com>
2022-07-15 09:31:34 +02:00
Tao Zhu f13b5c832d
Remove obsoleted metrics reporting to reduce lock contention on cost_model (#26608)
remove obsoleted metrics reporting to reduce lock contention on cost_model
2022-07-14 23:02:49 -05:00
HaoranYi 819d6512ab
add PartialEq, Eq and spell correction (#26614) 2022-07-14 14:55:03 +00:00
Jeff Washington (jwash) 3e9bd6170c
use OnceCell instead of RwLock for rent_paying_accounts_by_partition (#26601) 2022-07-14 08:52:41 -05:00
Jeff Washington (jwash) 4dea32e8e5
fix active_scans metric to not swap (#26602) 2022-07-13 20:11:23 -05:00
Jeff Washington (jwash) 2b7cc107d8
typo (#26618) 2022-07-13 17:55:02 -05:00
HaoranYi 1aa9215411
Refactor: add account_iter api for appendvec (#26591)
* add account iterator for appendvec

* semicolon
2022-07-12 23:56:05 +00:00
Jeff Washington (jwash) 6b0eb5a42b
handle rent paying accounts differently (#26487)
* handle rent paying accounts differently

* restore collection

* downgrade assert to metric and warn until we get more runtime
2022-07-12 17:51:33 -05:00
Brooks Prumo 1dfbb225e3
Fix test_accounts_data_size_and_rent_collection (#26576) 2022-07-12 17:05:21 -05:00
Jeff Washington (jwash) c62c55a16b
shrink stores by id (#26490) 2022-07-12 15:43:41 -05:00
Jeff Washington (jwash) 9d31216d23
add StoreReclaims::Ignore (#26489) 2022-07-12 15:05:47 -05:00
HaoranYi bf14440895
clean up and optimize account hash verify (#26560)
* remove unused code

* extract test related fault hash inject fn

* use rotate to optimize hashes removal

* use rotate to optimize snapshot hashes removal

* address code reveiw feedbacks

* revise comments

* inline
2022-07-12 19:27:28 +00:00
Jeff Washington (jwash) f54f528148
make bank test creation standard (#26519) 2022-07-12 09:07:55 -05:00
github-actions[bot] fd5df1cf25
Bump Version to 1.11.4 (#26578)
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
2022-07-11 23:30:38 -05:00
Jeff Washington (jwash) 1db136a8be
do not count 0 lamport accounts as rent-paying (#26514) 2022-07-11 22:16:46 -05:00
Jeff Washington (jwash) 36bd46bece
ancient append vec: reuse AccountsToStore (#26572) 2022-07-11 22:13:09 -05:00
Nicholas Clarke ee0a40937e
Add validator argument log_messages_bytes_limit to change log truncation limit.
Add new cli argument log_messages_bytes_limit to solana-validator to control how long program logs can be before truncation
2022-07-11 10:53:18 -05:00
behzad nouri ba785cf8ab
removes erroneous uses of std::mem::swap (#26536)
All instances should be replace by std::mem::{replace,take},
or just plain assignment.
2022-07-11 11:33:15 +00:00
Jeff Washington (jwash) 602da5e51f
add accounts db config to bank tests (#26517) 2022-07-10 19:42:06 -05:00
Jeff Washington (jwash) 2b6c9852f1
add more slop to test_age_time (#26537) 2022-07-10 17:52:07 -05:00
Jeff Washington (jwash) 1babf460ab
collapse some bank test creation fns (#26532) 2022-07-10 17:51:38 -05:00
behzad nouri 105c7e19e1
moves check for feature activation out of accounts loop (#26535) 2022-07-10 17:39:19 +00:00
Jeff Washington (jwash) 8576832f2a
rename Bank::new_with_config_for_tests (#26533) 2022-07-10 09:33:13 -05:00
Brooks Prumo 785a7a5936
Track more accounts data size changes (#26467) 2022-07-08 19:17:15 -05:00
behzad nouri c99d9f00a9
preserves rent_epoch for rent exempt accounts (#26479)
https://github.com/solana-labs/solana/pull/22292
prevents rent paying account creation going forward. As a result
rent_epoch field for rent exempt accounts is redundant, and advancing
this field will incur expensive account rewrites and cause discrepancy
between accounts-db and cached vote/stake accounts.

This commit adds a feature which upon activation preserves rent_epoch
field for rent exempt accounts so that the field is frozen and is no
longer advanced.
2022-07-08 20:04:08 +00:00
HaoranYi 995756a550
Use RefCount type alias instead of u64 (#26472)
use RefCount type alias instead of u64
2022-07-08 13:51:16 -05:00
Brooks Prumo 78b8a8a6db
Remove `get_total_resize_remaining()` and `total_resize_limit` from `TransactionContext` (#26504) 2022-07-08 20:26:54 +02:00
dependabot[bot] d1370f2c7d
chore: bump bytemuck from 1.9.1 to 1.10.0 (#26495)
* chore: bump bytemuck from 1.9.1 to 1.10.0

Bumps [bytemuck](https://github.com/Lokathor/bytemuck) from 1.9.1 to 1.10.0.
- [Release notes](https://github.com/Lokathor/bytemuck/releases)
- [Changelog](https://github.com/Lokathor/bytemuck/blob/main/changelog.md)
- [Commits](https://github.com/Lokathor/bytemuck/compare/v1.9.1...v1.10.0)

---
updated-dependencies:
- dependency-name: bytemuck
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-07-08 10:04:43 -06:00
Jeff Washington (jwash) d9eee72edb
remove FoundStoredAccount.account_size (#26460) 2022-07-07 18:37:14 -05:00
Brooks Prumo 3b759bcffb
Add fn to get max accounts data size from Bank (#26478) 2022-07-07 18:08:42 -05:00
Jeff Washington (jwash) 38d53ef8a2
add metrics for # scans active and scan distance (#26395) 2022-07-07 16:20:32 -05:00
Jeff Washington (jwash) ee0a54ce80
improve clean acct idx calls (#26320) 2022-07-07 15:50:12 -05:00
Jeff Washington (jwash) b582e4ce0f
introduce UpsertReclaim (#26462) 2022-07-07 15:40:17 -05:00
github-actions[bot] 9d937fb8a0
Bump Version to 1.11.3 (#26481)
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
2022-07-07 14:39:46 -05:00
Jeff Washington (jwash) c9f1d01e79
refactor, add GetUniqueAccountsResult (#26457) 2022-07-07 13:49:33 -05:00
Jeff Washington (jwash) 8e64b5883e
restore population of 'accounts' metric in rent collection (#26454) 2022-07-07 13:19:23 -05:00
Jeff Washington (jwash) 72256ac54d
use iter for append vecs (#26459) 2022-07-07 13:13:47 -05:00
Justin Starry 8416090732
Clean up `enable_durable_nonce` feature (#26470)
Clean up enable_durable_nonce feature
2022-07-07 17:14:46 +01:00
Justin Starry eb6a722eba
Clean up `nonce_must_be_writable` feature (#26444)
Clean up nonce_must_be_writable feature
2022-07-07 15:45:19 +01:00
Jeff Washington (jwash) b3a47de1ce
improve startup stats for rent payers (#26432) 2022-07-07 08:39:57 -05:00
Brooks Prumo f8bdedb596
Make accounts data size tests more robust (#26466) 2022-07-07 00:01:13 -05:00
Xiang Zhu 134303714b
Fix the order assumption of the pubkeys created by Pubkey::new_unique() (#26451)
new_unique() does not gurantee the increment order due to the bytes
array storage and its eq-partial trait interpreting the bytes in the
big-endian way.
2022-07-06 15:35:32 -07:00
carllin 90ef2cd02a
Parse snapshot for bank fields (#26016) 2022-07-06 17:30:30 -05:00
Jeff Washington (jwash) 16219e300e
generating index sets uncleaned_roots correctly (#26431)
* generating index sets uncleaned_roots correctly

* fix test failures

* rename

* update comments
2022-07-06 16:12:47 -05:00
behzad nouri d7201a8d1a
names fields in RentResullt::CollectRent enum variant (#26449)
Avoiding ambiguous raw tuple:
    CollectRent((Epoch, u64))

Using named fields instead:
    CollectRent {
        new_rent_epoch: Epoch,
        rent_due: u64,
    },
2022-07-06 20:01:16 +00:00
Justin Starry f4718be22f
Clean up `leave_nonce_on_success` feature (#26447)
Clean up leave_nonce_on_success feature
2022-07-06 17:59:34 +00:00
Justin Starry ff1e6fcbed
Clean up `nonce_must_be_advanceable` feature (#26446)
Clean up nonce_must_be_advanceable feature
2022-07-06 17:36:33 +00:00
Alexander Meißner 06ebfa1eb2
Replicates `AccountsDataMeter` in `TransactionContext` (#26438)
Replicates AccountsDataMeter in TransactionContext.
2022-07-06 19:27:42 +02:00
Justin Starry 611ac33718
Clean up `nonce_must_be_authorized` feature (#26445)
Clean up nonce_must_be_authorized feature
2022-07-06 17:17:31 +00:00
Xiang Zhu 44f499cff3
Refactor account index scan function parameters (#26428)
* Refactor account index scan function parameters

F: FnMut(bool, &SlotList<T>, &'a Pubkey, RefCount) -> AccountsIndexScanResult,
Refactor to take  (&'a Pubkey, Option<(&SlotList<T>, RefCount))

* Update comments based on review feedback

* cargo fmt --all fixes
2022-07-06 09:49:11 -07:00
Jeff Washington (jwash) 17a99d98dd
Revert "avoid adding to 'uncleaned_roots' when generating index and c… (#26441)
Revert "avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink (#25936)"

This reverts commit e24cc537a4.
2022-07-06 11:32:45 -05:00
Jeff Washington (jwash) ab164fc975
simplify code (#26443) 2022-07-06 11:22:26 -05:00
Justin Starry f8dccd4602
Clean up `max_tx_account_locks` feature (#26440)
Clean up max_tx_account_locks feature
2022-07-06 15:06:03 +00:00
Jeff Washington (jwash) 5afe4d938d
combine lookup in calc_delete_dependencies (#26429) 2022-07-06 08:50:14 -05:00
Jeff Washington (jwash) 71c7b4f9cb
use append vec iter in construct_candidate_clean_keys (#26426) 2022-07-06 08:49:35 -05:00
behzad nouri 6f5857a5db
removes feature gate code separating durable nonce from blockhash domain (#26055) 2022-07-06 12:03:13 +00:00
dependabot[bot] 37f4621c06
chore: bump serde from 1.0.137 to 1.0.138 (#26421)
* chore: bump serde from 1.0.137 to 1.0.138

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.137 to 1.0.138.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.137...v1.0.138)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-07-05 23:18:08 -06:00
Tao Zhu c1d89ad749
forward packets by prioritization in desc order (#25406)
- Forward packets by prioritization in desc order
- Add support of cost-tracking by transaction requested compute units
- Hook up account buckets to forwarder
- Add metrics for forwardable batches count
- Remove redundant invalid packets filtering at end of slot since forwarder will do the same when batch forwardable packets
- Add bench test for forwarding
2022-07-05 23:24:58 -05:00
Jeff Washington (jwash) 38216aa781
update comment for the next weary traveller (#26413) 2022-07-05 23:12:35 -05:00
Jeff Washington (jwash) 75149fd624
move mark_old_slots_as_dirty to be called from ahv (#26411) 2022-07-05 23:11:32 -05:00
Jeff Washington (jwash) 2aafef38f9
is_shrinking_productive calls avoid arc clone (#26422) 2022-07-05 17:53:29 -05:00
Jeff Washington (jwash) b7e34aea15
refactor calculate_rent_result (#26416)
reactor calculate_rent_result
2022-07-05 16:45:59 -05:00
Brooks Prumo 53b9420562
Remove `INTO!` macros in serde_snapshot (#26409) 2022-07-05 16:37:20 -05:00
carllin ce39c14025
Add end-to-end replay slot metrics (#25752) 2022-07-05 13:58:51 -05:00
Jeff Washington (jwash) 0ab521d5f1
inc counter when append vec drop fails (#26396) 2022-07-05 11:27:01 -05:00
Jeff Washington (jwash) 5f3b7bdd16
prevent ledger tool from calculating hash on non-rooted slots (#26355) 2022-07-02 11:50:01 -05:00
Jeff Washington (jwash) 88b5b7e30a
improve log (#26360) 2022-07-01 13:12:52 -05:00
HaoranYi 728986ddf4
Spell correction and add remove slot in AccountDB purge assert message (#26358)
nit spell. add remove slots in accoutdb purge assert message
2022-07-01 17:39:35 +00:00
Michael Vines fdb186ba3b pacify nightly clippy 2022-06-30 21:31:49 -07:00
Alexander Meißner 72a968fbe8
Fix/system instruction processor tests (#26338)
* Make mock_process_instruction() stricter by invoking the verify() method.

* Fixes broken test cases.
2022-07-01 00:06:24 +02:00
Jack May 4563bf40f6
cleanup feature: tx-wide-compute-cap (#26326) 2022-06-29 23:54:45 -07:00
Brooks Prumo 36d13af015
Add stats for Accounts Background Service (#26298)
* Add stats for Accounts Background Service

* pr
2022-06-30 02:21:30 +00:00
behzad nouri d053ce79d4
removes RwLock+Once in favor of OnceCell in caching staked-nodes (#26313)
VoteAccounts uses std::sync::{RwLock, Once} to lazily compute and cache
staked_nodes:
https://github.com/solana-labs/solana/blob/032bee13a/runtime/src/vote_account.rs#L89-L104

This commit instead switches to using once_cell::sync::OnceCell which
provides this exact intended functionality by design.
2022-06-29 22:22:22 +00:00
Jeff Washington (jwash) 557bf6e656
allow initial hash calc to occur in bg (#26271)
* allow initial hash calc to occur in bg

* validator_initialized -> startup_verification_complete

* add infos for leader and vote

* rework snapshot for startup verification

* change to assert
2022-06-29 16:48:33 -05:00
Jeff Washington (jwash) 608191352f
track # roots added/removed since last clean (#26319) 2022-06-29 16:47:57 -05:00
Jeff Washington (jwash) 2deb396490
remove noisy and useless 'error!' and add test (#26138) 2022-06-29 16:26:23 -05:00
Alexander Meißner c01a8f271e
Prepare built-in program tests to deal with stricter account modification checks (#26314)
* Fixes test_vote_state_withdraw().

* Set is_writable to true where needed.
2022-06-29 19:29:12 +02:00
Jeff Washington (jwash) e241906db2
remove outdated, confusing comment (#26315) 2022-06-29 10:42:34 -05:00
Jeff Washington (jwash) ca83f143aa
load_accounts_index_for_shrink uses scan (#26260) 2022-06-29 09:02:27 -05:00
behzad nouri af7f08eba4
uses OnceCell instead of RwLock+Once to cache vote-state in vote-account (#26257)
RwLock seems excessive since only the very 1st call to VoteAccount::vote_state
will write-lock the inner field. Future calls would also incur overhead of an
RwLockReadGuard.
once_cell::sync::OnceCell provides a matching api to the desired functionality.
2022-06-29 11:45:53 +00:00
Justin Starry 50631e274d
Ensure that staked nodes is populated before returning length (#26293) 2022-06-28 19:15:42 +00:00
Jeff Washington (jwash) 89e5640c63
AccountsIndex::Scan takes iter (#26268) 2022-06-28 12:47:17 -05:00
Jeff Washington (jwash) d07b079850
make test_age_time less intermittent (#26280) 2022-06-28 09:13:01 -05:00
Jeff Washington (jwash) bfc390f9ed
refactor acct idx scan func for reuse (#26252) 2022-06-28 09:01:04 -05:00
Jeff Washington (jwash) 4b591d7b59
AccountsDb::load_accounts_index_for_shrink takes slice (#26267) 2022-06-28 08:59:57 -05:00
Alexander Meißner e8fed88669
Both kinds of LamportsError are turned into InstructionError::ArithmeticOverflow anyway. (#26273) 2022-06-28 10:04:54 +02:00
Jeff Washington (jwash) e164879f20
load_accounts_index_for_shrink uses &item instead of (&item.0, &item.1) (#26266) 2022-06-27 22:40:45 -05:00
Jeff Washington (jwash) d6a28cb7a8
load_accounts_index_for_shrink unrefed_pubkeys is optional (#26265) 2022-06-27 17:23:46 -05:00
apfitzge 50eb9965ab
fix test_new_from_file_crafted_executable for m1 (#26009)
* fix test_new_from_file_crafted_executable for m1

* re-add old assert/comment, but disable for aarch64
2022-06-27 12:03:52 -05:00
Jeff Washington (jwash) d8516b5c88
comment and expand on test (#26231) 2022-06-27 11:35:46 -05:00
Brooks Prumo 662818ef0d
Use `VoteAccount::node_pubkey()` (#26207) 2022-06-27 09:09:06 -05:00
Jeff Washington (jwash) cbefc581c0
test_rent_collect_rent_from_accounts (#26204) 2022-06-25 19:04:28 -05:00
Justin Starry 0ca963a869
Refactor: Add separate methods for retrieving stake info from VoteAccounts (#26224) 2022-06-25 20:03:15 +00:00
Justin Starry 7cd7173b71
Refactor: Add get_delegated_stake method to VoteAccounts (#26221) 2022-06-25 16:41:35 +00:00
Justin Starry 44d1e62007
Refactor: No need to return stake in Bank::get_vote_account (#26220) 2022-06-25 16:27:43 +00:00
behzad nouri 2efdb965dd
encapsulates AuthorizeNonceAccount implementation in the nonce module (#26202)
Follow up commit removes feature gate code separating durable nonce from
blockhash domain. This commit allows to encapsulate any logic
distinguishing legacy vs current nonce versions in the nonce module
after removing the feature gate.
2022-06-25 12:17:39 +00:00
Jeff Washington (jwash) a3395a786a
vote_account uses AccountSharedData to avoid copies (#23687)
* vote_account uses AccountSharedData to avoid copies

* simpler deserialize
2022-06-24 15:08:01 -05:00
Nicholas Rempel 2b21c1da29 Sort Cargo.toml files 2022-06-24 12:41:38 -07:00
Jeff Washington (jwash) 18397527ad
retain prior metric behavior for invalid_cached_stake_accounts (#26181) 2022-06-24 14:12:16 -05:00
Jeff Washington (jwash) 054b25729d
eliminate vec creation in stake rewards (#26191) 2022-06-24 13:40:53 -05:00
HaoranYi fbcb4d8216
extract get_stake_reward fn from StakeReward (#26183) 2022-06-24 08:44:48 -05:00
Brooks Prumo 877fedadac
Remove StatusCacheRc type and use StatusCache directly (#26184) 2022-06-24 08:38:56 -05:00
apfitzge 4a729effff
Bugfix/test get rent collection accounts remove random account (#26178)
* Use default GenisisConfig so we don't have a random account

* Formatting
2022-06-24 09:22:50 -04:00
Jeff Washington (jwash) a36abfda28
don't skip rewrite when account was written IN the current slot already (#26197) 2022-06-24 07:28:37 -05:00
Brooks Prumo 5b842669e7
Remove InMemAccountsIndex::map() and use map_internal directly (#26189) 2022-06-23 15:55:35 -05:00
Brooks Prumo 23c50a2389
Add StatusCache::root_slot_deltas() and use it (#26170) 2022-06-23 15:19:06 -05:00
apfitzge 7efd05f7b3
Change get_pubkeys to match the new enum (#26177) 2022-06-23 11:56:09 -05:00
Boqin Qin(秦 伯钦) 355e09e1fb
runtime: fix possible deadlock in in_mem_accounts_index (#26046) 2022-06-23 11:35:47 -05:00
Jeff Washington (jwash) 102fb98070
impl StorableAccounts for [StakeReward] (#26165) 2022-06-23 08:34:28 -05:00
Jeff Washington (jwash) 19b79aa764
add metrics for invalid_cached_stake_accounts_rent_epoch (#26143) 2022-06-23 08:33:24 -05:00
Jeff Washington (jwash) 9db44ce352
cleanup some AccountSharedData::from(Account (#26120) 2022-06-23 08:32:48 -05:00
Brooks Prumo 7d29c26315
Remove auto traits from ICE work-around (#26098) 2022-06-23 07:22:30 -05:00
Brooks Prumo a5cf72e446
StatusCache::roots() returns `Slot`s not `u64`s (#26164) 2022-06-23 07:03:09 +00:00
github-actions[bot] 5c2f819f99
Bump Version to 1.11.2 (#26159) 2022-06-22 21:16:18 -05:00
Jeff Washington (jwash) aea75e82e3
Bank::store_accounts takes StorableAccounts (#26113) 2022-06-22 21:11:01 -05:00
Jeff Washington (jwash) a32523c43b
cleanup some tests (#26139) 2022-06-22 21:10:46 -05:00
HaoranYi c4efb9f19e
Stake cache size (#26071)
* reduce threadpool for stake processing

* wip

* Revert "reduce threadpool for stake processing"

This reverts commit 004a4f872ea7a3ef53e38d145b6350c3f57c680c.

* batch update stake_cache

* fix deadlock

* add test

* code review feedbacks

* more review feedbacks

* fix conflicts

* report stake account len and vote account len at epoch boundary

* report num_staked_nodes

* remove batch store, no atomic for redeem timing
2022-06-22 16:22:22 -05:00
Jeff Washington (jwash) c5c2b44ecc
adjust cached stake account compare for rent_epoch (#26135) 2022-06-22 15:11:25 -05:00
ivandzen 1ea6a2cb91
Extended interface for AccountOverrides (#25964)
* Extended interface for AccountOverrides

* fix checker issue

* make accounts private

Co-authored-by: Ivan Loboda <lobodatell@yandex.ru>
2022-06-22 15:22:31 -04:00
apfitzge f4189c0305
ledger-tool minimized snapshots (#25334)
* working on local snapshot

* Parallelization for slot storage minimization

* Additional clean-up and fixes

* make --minimize an option of create-snapshot

* remove now unnecessary function

* Parallelize parts of minimized account set generation

* clippy fixes

* Add rent collection accounts and voting node_pubkeys

* Simplify programdata_accounts generation

* Loop over storages to get slot set

* Parallelize minimized slot set generation

* Parallelize adding owners and programdata_accounts

* Remove some now unncessary checks on the blockstore

* Add a warning for minimized snapshots across epoch boundary

* Simplify ledger-tool minimize

* Clarify names of bank's minimization helper functions

* Remove unnecesary funciton, fix line spacing

* Use DashSets instead of HashSets for minimized account and slot sets

* Filter storages uses all threads instead of thread_pool

* Add some additional comments on functions for minimization

* Moved more into bank and parallelized

* Update programs/bpf/Cargo.lock for dashmap in ledger

* Clippy fix

* ledger-tool: convert minimize_bank_for_snapshot Measure into measure!

* bank.rs: convert minimize_bank_for_snapshot Measure into measure!

* accounts_db.rs: convert minimize_accounts_db Measure into measure!

* accounts_db.rs: add comment about use of minimize_accounts_db

* ledger-tool: CLI argument clarification

* minimization functions: make infos unique

* bank.rs: Add test_get_rent_collection_accounts_between_slots

* bank.rs: Add test_minimization_add_vote_accounts

* bank.rs: Add test_minimization_add_stake_accounts

* bank.rs: Add test_minimization_add_owner_accounts

* bank.rs: Add test_minimization_add_programdata_accounts

* accounts_db.rs: Add test_minimize_accounts_db

* bank.rs: Add negative case and comments in test_get_rent_collection_accounts_between_slots

* bank.rs: Negative test in test_minimization_add_programdata_accounts

* use new static runtime and sdk ids

* bank comments to doc comments

* Only need to insert the maximum slot a key is found in

* rename remove_pubkeys to purge_pubkeys

* add comment on builtins::get_pubkeys

* prevent excessive logging of removed dead slots

* don't need to remove slot from shrink slot candidates

* blockstore.rs: get_accounts_used_in_range shouldn't return Result

* blockstore.rs: get_accounts_used_in_range: parallelize slot loop

* report filtering progress on time instead of count

* parallelize loop over snapshot storages

* WIP: move some bank minimization functionality into a new class

* WIP: move some accounts_db minimization functionality into SnapshotMinimizer

* WIP: Use new SnapshotMinimizer

* SnapshotMinimizer: fix use statements

* remove bank and accounts_db minimization code, where possible

* measure! doesn't take a closure

* fix use statement in blockstore

* log_dead_slots does not need pub(crate)

* get_unique_accounts_from_storages does not need pub(crate)

* different way to get stake accounts/nodes

* fix tests

* move rent collection account functionality to snapshot minimizer

* move accounts_db minimize behavior to snapshot minimizer

* clean up

* Use bank reference instead of Arc. Additional comments

* Add a comment to blockstore function

* Additional clarifying comments

* Moved all non-transaction account accumulation into the SnapshotMinimizer.

* transaction_account_set does not need to be mutable now

* Add comment about load_to_collect_rent_eagerly

* Update log_dead_slots comment

* remove duplicate measure/print of get_minimized_slot_set
2022-06-22 13:17:43 -04:00
Jeff Washington (jwash) cf5ec13f59
Accounts::store_accounts_cached uses StorableAccounts (#26112) 2022-06-22 12:13:56 -05:00
Michael Vines f3639b76ce Remove some clippy lints 2022-06-22 09:23:22 -07:00
HaoranYi 5624ab0d4c
Optimize stake accounts store (#26004)
* reduce threadpool for stake processing

* wip

* Revert "reduce threadpool for stake processing"

This reverts commit 004a4f872ea7a3ef53e38d145b6350c3f57c680c.

* batch update stake_cache

* fix deadlock

* add test

* code review feedbacks

* more review feedbacks

* clippy

* update comments

* fix conflicts

* remove batch store, no atomic for redeem timing

* refactor stake_reward struct
2022-06-22 10:45:58 -05:00
Jeff Washington (jwash) 8e2bae7bf9
batch insert updates stat count correctly (#26107) 2022-06-22 08:45:42 -05:00
Jeff Washington (jwash) d3829b1a76
store_cached takes StorableAccounts to eliminate collect (#26109) 2022-06-22 08:45:03 -05:00
Brooks Prumo 8a18e02f80
Call StatusCache::append() directly (#26125) 2022-06-22 07:42:01 -05:00
Brooks Prumo 31c9b03049
Remove unused SignatureConfirmationStatus (#26124) 2022-06-22 07:41:22 -05:00
Brooks Prumo 97bd81a32b
Add bench for StatusCache::slot_deltas() (#26123) 2022-06-22 07:41:06 -05:00
Brooks Prumo d35c1a7d04
Use BankStatusCache in bank.rs (#26114) 2022-06-21 23:16:16 +00:00
Jeff Washington (jwash) 43e0d29b18
Revert "fix double count of account index del stats (#25797)" (#26103)
This reverts commit 05cb25d8da.
2022-06-21 14:05:26 -05:00
Jeff Washington (jwash) 7a3dc3a57b
remove unused param (#26096) 2022-06-21 13:10:41 -05:00
Will Hickey 51f26dc96e
Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
Boqin Qin(秦 伯钦) 95ea506d75
runtime: fix double-readlock in in_mem_accounts_index.rs (#26047) 2022-06-20 15:44:19 +01:00
Tyera Eulberg 752c851070
Clean up feature preventing new rent-paying accounts (#26000)
* Clean up require_rent_exempt_accounts feature

* Fix tests in bank_client, system_instruction_processor

* Fix tests in snapshot_utils

* Fix bank tests

* Ignore rent tests

* Fix stake-accounts tests
2022-06-16 15:35:25 -06:00