Brooks Prumo
95fbc29500
Add stats when getting snapshot storages ( #20234 )
2021-09-27 13:07:00 -05:00
Jeff Washington (jwash)
46263d405e
fix math error with stats calc ( #20253 )
2021-09-27 11:55:19 -05:00
Jeff Washington (jwash)
bc50fd4ad6
allocate threads for accounts idx based on cpus ( #20118 )
2021-09-27 11:25:06 -05:00
Jeff Washington (jwash)
025261c05e
AcctIdx: when no disk index, always wait for stats ( #20248 )
2021-09-27 11:24:36 -05:00
Jeff Washington (jwash)
833f7761d2
AcctIdx: upsert uses get from mem first ( #20186 )
2021-09-27 11:23:53 -05:00
Jeff Washington (jwash)
d8f98b3dcb
AcctIdx: avoid crazy #s on ms_per_age stat ( #20184 )
2021-09-27 08:25:23 -05:00
Alexander Meißner
df6905c3a6
Refactor: Merge MessageProcessor into InvokeContext ( #20165 )
...
* Inlines MessageProcessor::execute_instruction() in MessageProcessor::process_message().
* Moves MessageProcessor::create_pre_accounts() into ThisInvokeContext::push().
* Hoists ThisInvokeContext::new() out of loop inside MessageProcessor::process_message().
* Moves MessageProcessor::verify_account_references() and MessageProcessor::verify() into InvokeContext::verify().
2021-09-27 08:28:45 +02:00
Jeff Washington (jwash)
c6d189dac2
bg threads wait until next expected event ( #20126 )
2021-09-24 16:00:41 -05:00
Jeff Washington (jwash)
122206dbb1
AcctIdx: bg loop sleeps when all buckets flushed at current age ( #20168 )
2021-09-24 13:19:06 -05:00
Jeff Washington (jwash)
1d13594c1c
AcctIdx: store # threads in BucketHolder. Used later. ( #20151 )
2021-09-23 19:56:44 -05:00
Jeff Washington (jwash)
b3bb079d9f
AcctIdx: condense upsert to always use read lock ( #20148 )
2021-09-23 19:19:27 -05:00
Brooks Prumo
1347b507da
Handle duplicate AppendVec IDs ( #20096 )
...
When reconstructing the AccountsDb, if the storages came from full and
incremental snapshots generated on different nodes, it's possible that
the AppendVec IDs could overlap/have duplicates, which would cause the
reconstruction to fail.
This commit handles this issue by unconditionally remapping the
AppendVec ID for every AppendVec.
Fixes #17088
2021-09-23 17:34:32 -05:00
Jeff Washington (jwash)
2ae1e80013
AcctIdx: items needs to load from disk ( #20138 )
2021-09-23 17:33:54 -05:00
Jeff Washington (jwash)
334d38d200
AcctIdx: generation per thread starts at a random bucket ( #20122 )
2021-09-23 19:59:37 +00:00
Jeff Washington (jwash)
254d9c8903
range takes a ref ( #20140 )
2021-09-23 13:57:56 -05:00
Jeff Washington (jwash)
093dd68214
AcctIdx: stats for how much time bg threads wait ( #20127 )
2021-09-23 18:37:14 +00:00
Jeff Washington (jwash)
fe28f17718
AcctIdx: startup causes us to flush remove everything asap ( #20121 )
2021-09-23 12:17:52 -05:00
Jeff Washington (jwash)
5dae615208
AcctIdx: set age of cache accesses to future ( #20125 )
2021-09-23 11:43:48 -05:00
Jeff Washington (jwash)
fb55e9a9b6
AcctIdx: when stop_flush is disabled, prepare to age ( #20113 )
2021-09-23 11:42:32 -05:00
Jeff Washington (jwash)
5bbb0da7b8
AcctIdx: stats for in-mem ( #20114 )
2021-09-23 08:14:53 -05:00
Jeff Washington (jwash)
cb9d93525b
AcctIdx: metrics for loading from disk ( #20124 )
2021-09-23 08:14:08 -05:00
Jeff Washington (jwash)
4d3e32803f
AcctIdx: PreAllocatedAccountMapEntry ( #20117 )
2021-09-23 08:11:32 -05:00
Trent Nelson
7b365c564f
runtime: remove inactive delegation from stakes cache
2021-09-22 20:50:58 -06:00
Jeff Washington (jwash)
476124de50
AcctIdx: report disk bucket stats ( #20120 )
2021-09-22 18:17:20 -05:00
Jeff Washington (jwash)
f7eebadbed
AcctIdx: remove old items from cache ( #20076 )
2021-09-22 21:37:15 +00:00
Jeff Washington (jwash)
0eb0d7f73b
AcctIdx: consolidate lock_and_update_slot_list ( #20090 )
2021-09-22 18:40:19 +00:00
Jeff Washington (jwash)
4fb77183ef
AcctIdx: move background() to bucket holder ( #20085 )
2021-09-22 17:40:30 +00:00
Jeff Washington (jwash)
681e8728a1
get rid of logs left in test ( #20116 )
2021-09-22 12:39:48 -05:00
Jeff Washington (jwash)
bbd7bc2985
AcctIdx: Fix bad test ( #20115 )
2021-09-22 12:39:31 -05:00
Jeff Washington (jwash)
1eeddf2e23
orderings on AcctIdx dirty ( #20087 )
2021-09-22 11:24:16 -05:00
Jeff Washington (jwash)
681d6d121e
AcctIdx: 2 upserts use disk ( #20072 )
2021-09-22 11:14:08 -05:00
Jeff Washington (jwash)
deab944b30
AcctIdx: flush inserts to disk ( #20074 )
2021-09-22 09:55:40 -05:00
Jeff Washington (jwash)
804ed825f7
AcctIdx: rework timing of bg loop ( #20077 )
2021-09-22 09:39:49 -05:00
Jeff Washington (jwash)
a6aef76f52
AcctIdx: put_range_in_cache loads from disk ( #20073 )
2021-09-22 09:37:34 -05:00
Jeff Washington (jwash)
c944bfb1d9
AcctIdx tests use disk buckets ( #20070 )
2021-09-22 08:55:58 -05:00
Jeff Washington (jwash)
9e9417b474
AcctIdx: initialze stats with # bins ( #20088 )
2021-09-22 08:55:07 -05:00
Jeff Washington (jwash)
b918c2f4cd
AcctIdx: remove_if_slot_list_empty uses disk backing ( #20069 )
2021-09-22 08:48:44 -05:00
Jeff Washington (jwash)
6c518102dd
AcctIdx: fix some missing dirty calls ( #20089 )
2021-09-22 08:45:08 -05:00
dependabot[bot]
1b58166e9d
chore: bump memmap2 from 0.4.0 to 0.5.0 ( #20057 )
...
* chore: bump memmap2 from 0.4.0 to 0.5.0
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs ) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/RazrFalcon/memmap2-rs/releases )
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.4.0...v0.5.0 )
---
updated-dependencies:
- dependency-name: memmap2
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-21 17:20:49 -06:00
Jeff Washington (jwash)
e6795e6fdd
AcctIdx: get can load from disk ( #20067 )
2021-09-21 12:40:07 -05:00
Jeff Washington (jwash)
7f3d445af5
AcctIdx: Orderings and cleanup ( #20046 )
2021-09-21 10:52:39 -05:00
Jeff Washington (jwash)
bbe151e594
AcctIdx: only flush if disk index being used ( #20045 )
2021-09-21 08:42:13 -05:00
Jeff Washington (jwash)
38ea3aa31a
AcctIdx: keys impl when items are on disk ( #20039 )
2021-09-21 08:41:39 -05:00
Jeff Washington (jwash)
fb71e80962
AcctIdx: accessors for age ( #20023 )
2021-09-21 08:41:17 -05:00
Jeff Washington (jwash)
c3679ab9bd
AcctIdx: first pass at aging buckets ( #20008 )
2021-09-21 08:40:55 -05:00
Alexander Meißner
b507715d44
CPI without Account Refs ( #20034 )
...
* Removes search for accounts and unsafe lifetime transmute in InvokeContext::push().
* Replaces accounts by account_indices in verify_and_update() and process_cross_program_instruction().
Co-authored-by: Justin Starry <justin.m.starry@gmail.com>
2021-09-21 14:41:02 +02:00
Jeff Washington (jwash)
94330de843
AcctIdx: delete from disk index ( #20035 )
2021-09-20 16:19:21 -05:00
Jeff Washington (jwash)
6399a87485
AcctIdx: report age in stats ( #20025 )
2021-09-20 14:30:12 -05:00
Jeff Washington (jwash)
e9ee90a121
AcctIdx: age adds time component ( #20024 )
2021-09-20 14:29:34 -05:00
Jeff Washington (jwash)
d842705622
AcctIdx: when bg processing in multiple threads, avoid bin collisions ( #20019 )
2021-09-20 11:50:22 -05:00
Jeff Washington (jwash)
e6934e7247
AcctIdx: consolidate to correct CondVar ( #20017 )
2021-09-20 09:58:20 -05:00
Jeff Washington (jwash)
55d9ff9899
WaitableCondVar supports notify_one ( #20013 )
2021-09-20 09:57:57 -05:00
Jeff Washington (jwash)
bc7983b13e
AcctIdx: metric for active bg threads ( #20026 )
2021-09-20 09:44:40 -05:00
Jeff Washington (jwash)
2a42f8a06e
AcctIdx: disk index now holds a bucket map ( #19995 )
2021-09-20 08:40:10 -05:00
Jeff Washington (jwash)
af5b0d42a8
AcctIdx: initial index items have future age to flush ( #20010 )
2021-09-19 20:22:09 -05:00
Jeff Washington (jwash)
c1d181add5
add --accounts-index-memory-limit-mb ( #19269 )
2021-09-19 18:00:15 -05:00
Jeff Washington (jwash)
f579f73700
AcctIdx: introduce age per in-mem bucket ( #19982 )
2021-09-18 22:09:37 -05:00
Jeff Washington (jwash)
742155c214
call set_startup and add metrics on generate_index ( #20006 )
2021-09-18 22:08:58 -05:00
Jeff Washington (jwash)
db40d06a39
AcctIdx: introduce startup to bucket map ( #20004 )
2021-09-18 12:55:57 -05:00
Jeff Washington (jwash)
bed0049a51
AcctIdx: add age per item in in_mem acct idx ( #19981 )
2021-09-18 09:56:26 -05:00
Jeff Washington (jwash)
8df8f4396d
AcctIdx: start calling flush on in mem cache buckets ( #19966 )
...
* AcctIdx: start calling flush on in mem cache buckets
* fix orderings
2021-09-18 09:56:08 -05:00
Jeff Washington (jwash)
4089f8b06b
AcctIdx: with multiple bg threads, only log stats once per time interval ( #19984 )
2021-09-18 09:55:20 -05:00
Jeff Washington (jwash)
4e038e94fd
add 'drives' to AccountsIndexConfig ( #19989 )
2021-09-18 09:54:57 -05:00
Jeff Washington (jwash)
f500c99a6d
pass AcctIdxConfig to BucketMapHolder ( #19997 )
2021-09-18 09:54:00 -05:00
Jeff Washington (jwash)
bd86f41e18
acct idx tests use 2 bins instead of 8k ( #19996 )
2021-09-18 09:53:44 -05:00
Jeff Washington (jwash)
4dc2f08198
AcctIdx: hold_range_in_memory ( #19955 )
2021-09-17 22:19:29 +00:00
Jeff Washington (jwash)
24b136a993
AcctIdx: configurable flush threads ( #19983 )
2021-09-17 22:02:43 +00:00
Jeff Washington (jwash)
b2152be3b2
introduce bucket map ( #19848 )
...
* introduce bucket map
* rename BucketMap bits to num_buckets_pow2
* use u64::BITS
* Store the number of buckets in BucketMapConfig as a regular number
* remove redundant type aliases
* use Slot from sdk
* use Arc::clone() instead
* fixup erase drives
* rename num_buckets to max_buckets
* add doc to BucketMapConfig::new()
* add more documentation
* rename to DEFAULT_CAPACITY_POW2
* doc
* add more traits while we can
* rename capacity to capacity_pow2
* fix a naming for max_buckets_pow2
* remove unused/incorrect DataBucket::bytes
* rework benches a bit
* fixup bench docs
* rename create_bucket_capacity_pow2 to bucket_capacity_when_created_pow2
* rename BucketMapKeyValue to BucketItem
* rename to items_in_range
* remove values()
* remove addref and unref
* remove more addref and unref
* resurect addref and unref since tests use 'em for now
* rename to BucketStorage
* move stats in bucket_stats
* remove specializations (i don't think they are needed)
* move MaxSearch and RefCount into lib.rs
* move BucketItem to bucket_item.rs
* add doc
* keys no longer returns an option
* Revert "remove specializations (i don't think they are needed)"
This reverts commit b22f78e072cf0f7107851b08e58c2e3fead3f64d.
Co-authored-by: Brooks Prumo <brooks@solana.com>
2021-09-17 15:11:27 -05:00
Jeff Washington (jwash)
cddb9da4f0
AcctIdx: central age functions/state ( #19980 )
2021-09-17 15:11:07 -05:00
Jeff Washington (jwash)
99f5684dc4
AcctIdx: handle future config defaults ( #19967 )
2021-09-17 13:12:06 -05:00
Jeff Washington (jwash)
9998e16df3
AcctIdx: prepare to spin up multiple bg threads ( #19969 )
2021-09-17 11:53:25 -05:00
Jack May
7b0bf64404
cleanup old features ( #19956 )
2021-09-17 09:46:49 -07:00
Jeff Washington (jwash)
37d6f5013f
AcctIdx: initialize acct idx by config ( #19968 )
2021-09-17 16:39:41 +00:00
dependabot[bot]
b8106e2450
chore: bump memmap2 from 0.3.1 to 0.4.0 ( #19974 )
...
* chore: bump memmap2 from 0.3.1 to 0.4.0
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs ) from 0.3.1 to 0.4.0.
- [Release notes](https://github.com/RazrFalcon/memmap2-rs/releases )
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.3.1...v0.4.0 )
---
updated-dependencies:
- dependency-name: memmap2
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-09-17 10:00:12 -06:00
Jeff Washington (jwash)
376b20c3d2
AcctIdx: add some bucket flush helpers ( #19965 )
2021-09-17 10:41:30 -05:00
Jeff Washington (jwash)
4d8b3aa578
AcctsIdx: introduce dirty state per pubkey ( #19960 )
2021-09-16 21:36:23 -05:00
Jeff Washington (jwash)
58f25a8752
AcctIdx: move background() to AccountsIndexStorage ( #19948 )
2021-09-16 17:52:06 -05:00
Jeff Washington (jwash)
66e0fafc21
AccountsIndexStorage holds InMemAccountsIndex[] ( #19947 )
2021-09-16 21:12:22 +00:00
sakridge
dc69cc1ae4
Only allow votes when root distance gets too high ( #19917 )
2021-09-16 15:12:26 +02:00
Jack May
bd434d92e1
nit: misspelling ( #19929 )
2021-09-16 01:27:26 +00:00
dependabot[bot]
fd60ef78eb
chore: bump flate2 from 1.0.21 to 1.0.22 ( #19863 )
...
* chore: bump flate2 from 1.0.21 to 1.0.22
Bumps [flate2](https://github.com/rust-lang/flate2-rs ) from 1.0.21 to 1.0.22.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases )
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.0.21...1.0.22 )
---
updated-dependencies:
- dependency-name: flate2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-15 14:30:30 -06:00
Jeff Washington (jwash)
eddd583cd5
AcctIdx: pass bins to BucketMapHolder ( #19906 )
2021-09-15 18:07:53 +00:00
Jeff Washington (jwash)
99f2c746d1
AcctIdx: Put InMemAcctIndex behind arc ( #19904 )
2021-09-15 17:36:08 +00:00
Jeff Washington (jwash)
b467e7fb3b
AcctIdx: remove auto Debug trait ( #19905 )
2021-09-15 09:54:16 -05:00
Brooks Prumo
8e3c420414
Add logging after taking a bank snapshot ( #19891 )
2021-09-15 07:39:21 -05:00
Jeff Washington (jwash)
d3c80be7d5
AcctIdx: use stats() accessor ( #19890 )
2021-09-15 05:28:44 +00:00
Jeff Washington (jwash)
691bea8776
extract map to map() ( #19883 )
2021-09-14 23:08:17 -05:00
Jeff Washington (jwash)
229b378ed1
in-mem acct idx hash map uses rwlock ( #19878 )
2021-09-14 18:01:34 -05:00
Jeff Washington (jwash)
fc647eed19
introduce trait IndexValue for accounts index ( #19859 )
2021-09-14 17:51:07 -05:00
Brooks Prumo
21bc7e94f5
Add logging of incremental snapshots to purge_old_snapshot_archives() ( #19874 )
2021-09-14 17:49:15 -05:00
Jeff Washington (jwash)
60276da1fd
rework acct idx entry to prepare for hashmap ( #19875 )
2021-09-14 15:27:23 -05:00
Jeff Washington (jwash)
2a795614d3
accounts index keys returns by value ( #19858 )
2021-09-14 09:38:59 -05:00
Jack May
00d7981f64
Fix native invoke writable privileges ( #19750 )
...
* Fix native invoke writable privileges
* build downstream spl bpf programs for tests
2021-09-13 22:57:37 -07:00
Jeff Washington (jwash)
910f241c3f
add <T> through accounts index ( #19852 )
2021-09-13 22:59:03 -05:00
Jeff Washington (jwash)
b57e86abf2
cache account hash info ( #19426 )
...
* cache account hash info
* ledger_path -> accounts_hash_cache_path
2021-09-13 20:39:26 -05:00
Jeff Washington (jwash)
dca49a614f
size and per-bin stats ( #19842 )
2021-09-13 19:26:49 -05:00
carllin
87a7f00926
Track reset bank in PohRecorder ( #19810 )
2021-09-13 16:55:35 -07:00
Brooks Prumo
3071c4caad
Set is_startup=true for exhaustively_free_unused_resource() ( #19846 )
2021-09-13 18:08:06 -05:00
Jeff Washington (jwash)
e1a038e791
add stats inserts, deletes, updates_in_mem ( #19807 )
2021-09-13 15:59:01 -05:00
Brooks Prumo
ef749a2506
Flush accounts cache before calling clean_accounts() ( #19839 )
2021-09-13 14:56:13 -05:00