125f9634fd
add hash calc config.use_write_cache ( #24005 )
2022-03-30 17:19:34 -05:00
82c5230bc2
AccountsPackage::new less brittle ( #23968 )
2022-03-30 14:06:15 -05:00
1fb82d7924
fix typo in comments ( #24004 )
2022-03-30 09:47:51 -05:00
af9344fe22
SortedStorages refactoring ( #23998 )
2022-03-30 09:19:03 -05:00
5636570d6d
add roots_original to roots tracker ( #23849 )
2022-03-30 08:52:45 -05:00
da844d7be5
refactoring of SortedStorages tests to make other changes easier ( #23990 )
2022-03-29 22:06:48 -05:00
5a613e9b6e
use CalcAccountsHashConfig in calculate_accounts_hash ( #23987 )
2022-03-29 22:05:47 -05:00
794645d092
Adds check_number_of_instruction_accounts() to all builtin programs except for the address-lookup-table. ( #23984 )
2022-03-29 19:06:50 +02:00
31b707b625
Specify if archive size datapoint is for full or incremental snapshots ( #23941 )
2022-03-26 12:29:13 -05:00
c24de17278
remove index hash calculation as an option ( #23928 )
2022-03-25 15:32:53 -05:00
ec78702bc8
RollingBitField::get_all_less_than ( #23919 )
2022-03-25 15:20:22 -05:00
acfd22712b
RollingBitFIeld to its own file ( #23917 )
2022-03-25 10:37:00 -05:00
55d61023f7
document 'accounts' hash ( #23907 )
2022-03-24 15:58:52 -05:00
140c8dd01f
Refactor: Replaces KeyedAccount in_get_sysvar_with_account_check ( #23905 )
...
* Replaces all use sites of get_sysvar_with_account_check by get_sysvar_with_account_check2.
* Removes get_sysvar_with_account_check.
* Renames get_sysvar_with_account_check2 to get_sysvar_with_account_check.
2022-03-24 19:30:42 +01:00
37c36ce3fa
pass stats separately from CalcAccountsHashConfig ( #23892 )
2022-03-24 12:48:47 -05:00
82328fd9d8
move max_clean_root deeper in flush cache ( #23869 )
2022-03-24 12:45:49 -05:00
c31db81ac4
Use VoteAccountsHashMap type alias in all applicable spots ( #23904 )
2022-03-24 12:09:48 -05:00
a22a2384bf
fix ci test error ( #23908 )
2022-03-24 11:30:20 -05:00
5b916961b5
HashCalc uses self.accounts_cache ( #23890 )
2022-03-24 10:34:28 -05:00
f2aea3b7c7
flush_slot_cache takes [Slot] ( #23865 )
2022-03-24 10:24:36 -05:00
9d3b17c635
HashCalc uses self.accounts_index ( #23888 )
2022-03-24 10:06:32 -05:00
396b49a7c1
Start saving/loading prior_roots(_with_hash) to snapshot ( #23844 )
...
* Start saving/loading prior_roots(_with_hash) to snapshot
* Update runtime/src/accounts_index.rs
Co-authored-by: Michael Vines <mvines@gmail.com>
* Update runtime/src/accounts_index.rs
Co-authored-by: Michael Vines <mvines@gmail.com>
* update comment
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-03-24 10:06:24 -05:00
b22165ad69
hash calc uses self.filler_account_suffix ( #23887 )
2022-03-24 09:58:06 -05:00
9022931689
calc hash uses self.num_hash_scan_passes ( #23883 )
2022-03-24 09:44:42 -05:00
e3eb002f66
Log storage size stats at hash calc ( #23843 )
2022-03-24 09:40:35 -05:00
f1a411c897
add epoch_schedule and rent_collector to hash calc ( #23857 )
2022-03-24 09:39:22 -05:00
db5d68f01f
HashCalc uses self.accounts_hash_cache_path ( #23882 )
2022-03-24 09:31:55 -05:00
90009f330b
small refactor to shorten the lock on slot_under_contention hashset ( #23891 )
...
* small refactor to shorten the lock on slot_under_contention hashset
* adding comments
* comments
2022-03-24 08:20:56 -05:00
91c2729856
Replaces keyed_account get_signers() by InstructionContext::get_signers(). ( #23863 )
2022-03-24 12:57:51 +01:00
5a892af2fe
disable 'check_hash' on accounts hash calc ( #23873 )
2022-03-23 21:03:31 -05:00
3e22d4b286
calc hash uses self.thread_pool_clean ( #23881 )
2022-03-23 20:52:38 -05:00
486f7b7673
use array access function ( #23895 )
2022-03-23 17:03:01 -07:00
dc3863ef14
flush_slot_cache_with_clean ( #23868 )
2022-03-23 14:09:56 -05:00
260f899eda
write cache: hashmap to set ( #23866 )
2022-03-23 14:05:45 -05:00
9e61fe7583
add AccountsHashConfig to manage parameters ( #23850 )
2022-03-23 13:44:23 -05:00
b1280b670a
calculate_accounts_hash_without_index takes &self ( #23846 )
...
* calculate_accounts_hash_without_index takes &self
* Update runtime/src/snapshot_package.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-03-23 11:57:32 -05:00
7b89222fde
don't start extra threads for shrink/clean/hash ( #23858 )
2022-03-23 11:53:37 -05:00
493a8e2348
remove random flushing of write cache ( #23845 )
2022-03-23 08:45:44 -05:00
bc35e1c5f5
snapshot code needs all storages for hash calc ( #23840 )
2022-03-22 21:27:54 -05:00
9f0ca6d88a
Refactor: Remove `trait` from nonce keyed account ( #23811 )
...
* Removes the trait `NonceKeyedAccount`.
2022-03-23 02:09:30 +01:00
7af48465fa
transaction-status: Add return data to meta ( #23688 )
...
* transaction-status: Add return data to meta
* Add return data to simulation results
* Use pretty-hex for printing return data
* Update arg name, make TransactionRecord struct
* Rename TransactionRecord -> ExecutionRecord
2022-03-22 23:17:05 +01:00
1089a38aaf
AcctIdx: rework scan and write to disk ( #23794 )
2022-03-22 11:54:12 -05:00
89ba3ff139
log fail to evict ( #23815 )
2022-03-22 09:19:38 -05:00
c4ecfa5716
Bump version to v1.11 ( #23807 )
...
* Revert crossbeam_epoch to stable. 0.9.8 only works with nightly
* Remove unneeded unit expression
2022-03-21 17:40:50 -05:00
24f6855f86
AcctIdx: only remove a fixed number of items per write lock ( #23795 )
2022-03-21 16:55:04 -05:00
cb06126388
Set accounts_data_len on feature activation ( #23730 )
2022-03-21 12:28:26 -05:00
965ab9186d
AcctIdx: fix infinite loop ( #23806 )
2022-03-21 10:58:36 -05:00
15357480ec
Refactor instruction compilation and update message account key ordering ( #23729 )
...
* Refactor: Make instruction compilation usable for other message versions
* apply trents feedback
* Fix tests
* Fix bpf compatiblity
2022-03-21 20:53:32 +08:00
258db77100
AcctIdx: factor 'scan' out of flush_internal ( #23777 )
2022-03-20 22:00:38 -05:00
f34434f96b
Drop lock ( #23765 )
2022-03-20 21:27:24 -04:00
dd69f3baf5
throttle index adding to allow disk flushing to keep up and reduce startup ram usage ( #23773 )
2022-03-20 19:56:20 -05:00
335c4b668b
Fix bug in bank/sysvar_cache tests ( #23780 )
2022-03-19 21:38:18 -05:00
df29276eb0
AcctIdx: remove -> evict ( #23775 )
2022-03-18 17:13:21 -05:00
a419374fa4
factor out function ( #23742 )
2022-03-18 14:10:52 -05:00
56428be629
Not exposing inner cost_table to encapsulating implementation details,
...
making future change easier.
2022-03-18 12:58:43 -05:00
998e7d18f9
AcctIdx: never retry a bucket flush ( #23732 )
2022-03-18 12:20:42 -05:00
0ed23899e7
directly use compute_budget MAX_UNITS and DEFAULT_UNITS
2022-03-18 08:53:11 -05:00
a4cacf3389
add deterministic default cost
2022-03-18 08:53:11 -05:00
857576d76f
AcctIdx: move write to disk outside in mem write lock ( #23731 )
2022-03-17 23:09:41 -05:00
7ff8c80e25
Add accounts_data_len to bank snapshot ( #23714 )
2022-03-17 20:14:54 -05:00
664deb2157
AcctIdx: get rid of unused is_dirty ( #23733 )
2022-03-17 16:29:36 -05:00
2f58c9e501
Bump version to 1.10.4 ( #23743 )
2022-03-17 14:02:13 -05:00
66b1f55351
AcctIdx: cheaper check for should evict while flushing ( #23705 )
2022-03-17 08:46:32 -05:00
3a46f45650
AcctIdx: add stats for flushing and estimated memory ( #23709 )
2022-03-17 08:46:00 -05:00
b4350a2522
Make solana-address-lookup-table-program crate bpf compatible ( #23700 )
2022-03-17 08:21:07 +08:00
be0aeea01a
AcctIdx: check for range holds outside lock ( #23706 )
2022-03-16 17:44:59 -05:00
caddb851be
acct idx flush keeps slot_list read lock open ( #23704 )
2022-03-16 17:29:04 -05:00
fa7926580a
minor VoteAccount refactoring ( #23686 )
2022-03-16 09:50:12 -05:00
3252dc7203
uses structural sharing for stake-delegations hash-map ( #23585 )
...
StakeDelegations is using Arc to implement copy-on-write semantics:
https://github.com/solana-labs/solana/blob/58c0db970/runtime/src/stake_delegations.rs#L14-L16
However a single delegation change will still clone the entire hash-map,
resulting in excessive memory use as observed in:
https://github.com/solana-labs/solana/issues/23061#issuecomment-1063444072
This commit instead uses immutable hash-map implementing structural
sharing:
> which means that if two data structures are mostly copies of each
> other, most of the memory they take up will be shared between them.
https://docs.rs/im/latest/im/
2022-03-16 12:58:05 +00:00
584ac80b1e
Revert: `KeyedAccount` refactoings in builtin programs ( #23649 )
...
* Revert "Replaces KeyedAccount by BorrowedAccount in the BPF loader. (#23056 )"
6c56eb9663#23217 )"
ee7e411d68#23214 )"
1a68f81f89#23302 )"
a14c7c37ee#23348 )"
e2fa6a0f7a#23375 )"
ee3fc39f1c
2022-03-16 11:30:01 +01:00
2d3501dff9
make upsert infallible op
2022-03-15 17:05:41 -05:00
b5a99b9b09
avoid data copies in writing to cache ( #23674 )
2022-03-15 16:42:26 -05:00
29af42f428
chore: bump zstd from 0.11.0+zstd.1.5.2 to 0.11.1+zstd.1.5.2 ( #23679 )
...
* chore: bump zstd from 0.11.0+zstd.1.5.2 to 0.11.1+zstd.1.5.2
Bumps [zstd](https://github.com/gyscos/zstd-rs ) from 0.11.0+zstd.1.5.2 to 0.11.1+zstd.1.5.2.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/commits )
---
updated-dependencies:
- dependency-name: zstd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2022-03-15 15:31:14 -06:00
8c4f010b8d
fix logging: only walk remote dir if it exist ( #23663 )
2022-03-15 08:56:22 -05:00
8c8f9694e0
Refactor: Sanitized transaction creation ( #23558 )
...
* Refactor: SanitizedTransaction::try_create optionally computes hash
* Refactor: Add SimpleAddressLoader
2022-03-15 12:02:22 +08:00
f05ac7a899
move to get_slots_in_epoch ( #23657 )
2022-03-14 22:47:45 -05:00
102dd68a03
Rename AccountsDb plugins to Geyser plugins ( #23604 )
2022-03-14 19:18:46 -06:00
c68c0e881e
consolidate use-jit flag ( #23652 )
2022-03-14 15:00:00 -07:00
0c684721d8
Separate remotely downloaded snapshot archives ( #23510 )
...
* seperate remotely downloaded snapshot archives
* add str const for snapshot download dir
* only walk remote sub directory
* move directory creation outside of loop
* move is_remote to traits
* clippy simplify
* clippy
* clippy
* add unittest
* fix local cluster tests
* look for remote snapshot archive in remote foler
* create remote dir in tests
* use snapshot download dir constant
* extract build_remote_dir fn
* fix build
* code review - walking snapshot archived dirs explicitly
* fix build
* fix build
* fix comments
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* clippy
* borrow to avoid copy
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-03-14 14:03:59 -05:00
a3242b6b86
remove unnecessary namespaces ( #23646 )
2022-03-14 12:28:55 -05:00
63bf0f66af
Bump version to 1.10.3 ( #23648 )
2022-03-14 11:18:45 -05:00
11be3fb7fa
Add FlushGuard to ensure `flushing_active` is used safely ( #23571 )
2022-03-14 10:35:00 -05:00
e8a8f4e9e2
fix filler acct slots in epoch ( #23536 )
2022-03-14 09:57:44 -05:00
7758c32035
Banking Stage drops transactions that'll exceed the total account data size limit ( #23537 )
2022-03-13 15:58:57 +00:00
ccff006948
Add more logs to `purge_old_snapshot_archives` ( #23531 )
2022-03-11 15:01:48 -06:00
d20dd21600
Sort tables in Cargo.toml files ( #23602 )
2022-03-11 11:05:57 -06:00
9b591286d7
Revert "chore: bump dashmap from 4.0.2 to 5.1.0 ( #23372 )" ( #23592 )
...
This reverts commit 3a0271c113
2022-03-11 09:12:27 +01:00
e6a67cd091
Remove unused dependencies ( #23595 )
2022-03-10 22:20:52 -06:00
631be1ffdd
chore: bump zstd from 0.10.0+zstd.1.5.2 to 0.11.0+zstd.1.5.2 ( #23581 )
...
* chore: bump zstd from 0.10.0+zstd.1.5.2 to 0.11.0+zstd.1.5.2
Bumps [zstd](https://github.com/gyscos/zstd-rs ) from 0.10.0+zstd.1.5.2 to 0.11.0+zstd.1.5.2.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/compare/v0.10.0...v0.11.0 )
---
updated-dependencies:
- dependency-name: zstd
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-03-10 20:01:05 -07:00
b444836a97
Bump version to 1.10.2 ( #23597 )
2022-03-10 16:41:06 -06:00
3c6840050c
Ensure blocks do not exceed the max accounts data size during Replay Stage ( #23422 )
2022-03-10 10:24:31 -06:00
17b00ad3a4
Add quic-client module ( #23166 )
...
* Add quic-client module to send transactions via quic, abstracted behind the TpuConnection trait (along with a legacy UDP implementation of TpuConnection) and change thin-client to use TpuConnection
2022-03-09 21:33:05 -05:00
1fe0d6eeeb
Set ordering flushing_active.swap() to AcqRel ( #23567 )
2022-03-10 01:22:42 +00:00
1eddb6d1e9
Move ArchiveFormat into own module ( #23562 )
2022-03-09 16:09:34 -06:00
9bbccbe27c
Use `AsRef<Path>` instead of `PathBuf` for parameters ( #23560 )
2022-03-09 16:08:33 -06:00
ba54b30101
clippy ( #23563 )
2022-03-09 15:33:50 -06:00
5a0cd05866
Revert "- estimate a program cost as 2 standard deviation above mean"
...
This reverts commit a25ac1c988
2022-03-08 17:18:44 -08:00
9acbfa5eb1
Revert "use EMA in place of Welford"
...
This reverts commit 6587dbfa47
2022-03-08 17:18:44 -08:00
c878c9e2cb
Revert "1. Persist to blockstore less frequently;"
...
This reverts commit 7aa1fb4e24
2022-03-08 17:18:44 -08:00
cc4d75a16f
fix typo ( #23535 )
2022-03-08 18:28:00 -06:00
Jeff Washington (jwash)
HaoranYi
Alexander Meißner
Brooks Prumo
steviez
Jack May
Jon Cinque
Will Hickey
Justin Starry
carllin
Tao Zhu
behzad nouri
dependabot[bot]
Tyera Eulberg
Giorgio Gambino
sakridge
ryleung-solana