Commit Graph

2091 Commits

Author SHA1 Message Date
Justin Starry 0ddb34a0b4
Add struct and convenience methods to track stake activation status (#20392)
* Add struct and convenience methods to track stake activation status

* fix nits

* rename
2021-10-04 18:59:11 -04:00
Jeff Washington (jwash) b6a6d6986d
lift store.slot() from loop (#20409) 2021-10-04 16:07:01 -05:00
Jeff Washington (jwash) 8da2eb980a
AcctIdx: introduce BucketApi for access to a specific bucket (#20359) 2021-10-04 11:48:09 -05:00
Jeff Washington (jwash) e31c065544
AcctIdx: when disk index insert needs to grow, drop locks, grow, then restart flush (#20333) 2021-09-30 19:27:09 -04:00
Lijun Wang fe97cb2ddf
AccountsDb plugin framework (#20047)
Summary of Changes

Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows

Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.

The code comprises 4 major parts:

accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-09-30 14:26:17 -07:00
dependabot[bot] 9ef1f06b3f
chore: bump ouroboros from 0.11.1 to 0.12.0 (#20344)
* chore: bump ouroboros from 0.11.1 to 0.12.0

Bumps [ouroboros](https://github.com/joshua-maros/ouroboros) from 0.11.1 to 0.12.0.
- [Release notes](https://github.com/joshua-maros/ouroboros/releases)
- [Commits](https://github.com/joshua-maros/ouroboros/commits)

---
updated-dependencies:
- dependency-name: ouroboros
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-30 13:20:05 -06:00
Jeff Washington (jwash) 8db9586599
AcctIdx: randomly evict items from in mem (#20319) 2021-09-30 02:51:54 +00:00
Jeff Washington (jwash) 5e05f12c48
AcctIdx: bucket perf improvements (#20328) 2021-09-30 01:53:26 +00:00
Jeff Washington (jwash) 59112adac1
AcctIdx: us stats were showing ms (#20324) 2021-09-29 22:18:07 +00:00
Jeff Washington (jwash) 2ec74474e9
AcctIdx: optimize should_remove_from_mem (#20327) 2021-09-29 16:41:54 -05:00
Alexander Meißner 4de5fff3ca
Simplifies return_data accessors in InvokeContext. (#20290) 2021-09-29 19:11:06 +02:00
Alexander Meißner 57c8abf499
Refactor: Merge message processor into invoke context (#20308)
* Inlines MessageProcessor::execute_instruction() in MessageProcessor::process_message().

* Moves MessageProcessor::create_pre_accounts() into ThisInvokeContext::push().

* Move instruction_recorders slice into InvokeContext.

* Makes account_indices optional in InvokeContext::push().

* Separates initial InvokeContext::push() from ThisInvokeContext::new().

* invoke_context.pop() the base invocation frame.

* Zip message.instructions.iter() and program_indices.iter().

* Moves ThisInvokeContext::new() to the beginning of the loop inside MessageProcessor::process_message().

* Hoists ThisInvokeContext::new() out of loop inside MessageProcessor::process_message().

* Removes unnecessary clone() from ThisInvokeContext::new() in MessageProcessor ::process_message().

* Stop ignoring errors from MessageProcessor::create_pre_accounts().

* Moves MessageProcessor::verify_account_references() and MessageProcessor::verify() into InvokeContext::verify().
2021-09-29 12:05:25 -05:00
Jeff Washington (jwash) 062d8e6c5a
AcctIdx: fix logic on should_remove_from_mem (#20302) 2021-09-29 13:07:40 +00:00
Jeff Washington (jwash) bd9b2f6f39
AcctIdx: throttling bg flushing to spread out work over interval (#20176)
* AcctIdx: throttling bg flushing to spread out work over interval

* prevent hang
2021-09-29 07:42:54 -05:00
Jack May 8fee9a2e1a
Dont call precompiled programs (#19930) 2021-09-29 06:25:08 +00:00
Jack May 491877de3d
Revert "Refactor: Merge MessageProcessor into InvokeContext (#20165)" (#20301)
This reverts commit df6905c3a6.
2021-09-28 16:59:01 -07:00
Jeff Washington (jwash) fa5b091b4c
prior to panicing with cap mismatch, try other calculation (#20292) 2021-09-28 16:09:36 -05:00
Jeff Washington (jwash) 320beb76f6
AcctIdx: implement wait_for_idle (#20256) 2021-09-28 11:07:19 -05:00
Tyera Eulberg 2cd9dc99b6
Restore ability for programs to upgrade themselves (#20265)
* Make helper associated fn

* Add feature definition

* Add handling to preserve program-id write lock when upgradeable loader is present; restore bpf upgrade-self test

* Use single feature
2021-09-28 09:59:08 -06:00
Jack May 4e27543415
Allow programs to realloc their accounts within limits (#19475) 2021-09-28 01:13:03 -07:00
Jeff Washington (jwash) e50a26e6cf
AcctIdx: add age_interval_ms() to remove constant use (#20264) 2021-09-28 01:22:07 +00:00
Jeff Washington (jwash) af309c126c
AcctIdx: refactor get to use more refs (#20268) 2021-09-28 00:18:36 +00:00
dependabot[bot] 34406d4815
chore: bump ouroboros from 0.10.1 to 0.11.1 (#20244)
* chore: bump ouroboros from 0.10.1 to 0.11.1

Bumps [ouroboros](https://github.com/joshua-maros/ouroboros) from 0.10.1 to 0.11.1.
- [Release notes](https://github.com/joshua-maros/ouroboros/releases)
- [Commits](https://github.com/joshua-maros/ouroboros/commits)

---
updated-dependencies:
- dependency-name: ouroboros
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-27 23:26:08 +00:00
Jeff Washington (jwash) f6812523cd
handle race condition with flush and delete (#20255) 2021-09-27 16:55:15 -05:00
Jeff Washington (jwash) bbd2f9672d
clone -> Arc::clone to make cost clear (#20262) 2021-09-27 16:49:53 -05:00
Jeff Washington (jwash) 4f83818fdb
get rid of WriteAccountMapEntry instance (#19093) 2021-09-27 16:24:19 -05:00
Jeff Washington (jwash) e4795ae7ed
AcctIdx: keep cached or multiple slot lists in cache (#20257) 2021-09-27 16:09:26 -05:00
Jeff Washington (jwash) 9d91a6f1b7
shrink slots at startup uses more threads (#20251) 2021-09-27 14:14:55 -05:00
Jeff Washington (jwash) 8266864adc
AcctIdx: maybe advance age when a bg thread wakes up (#20250)
* fix math error with stats calc

* AcctIdx: maybe report stats when a bg thread wakes up
2021-09-27 18:39:41 +00:00
Brooks Prumo 95fbc29500
Add stats when getting snapshot storages (#20234) 2021-09-27 13:07:00 -05:00
Jeff Washington (jwash) 46263d405e
fix math error with stats calc (#20253) 2021-09-27 11:55:19 -05:00
Jeff Washington (jwash) bc50fd4ad6
allocate threads for accounts idx based on cpus (#20118) 2021-09-27 11:25:06 -05:00
Jeff Washington (jwash) 025261c05e
AcctIdx: when no disk index, always wait for stats (#20248) 2021-09-27 11:24:36 -05:00
Jeff Washington (jwash) 833f7761d2
AcctIdx: upsert uses get from mem first (#20186) 2021-09-27 11:23:53 -05:00
Jeff Washington (jwash) d8f98b3dcb
AcctIdx: avoid crazy #s on ms_per_age stat (#20184) 2021-09-27 08:25:23 -05:00
Alexander Meißner df6905c3a6
Refactor: Merge MessageProcessor into InvokeContext (#20165)
* Inlines MessageProcessor::execute_instruction() in MessageProcessor::process_message().

* Moves MessageProcessor::create_pre_accounts() into ThisInvokeContext::push().

* Hoists ThisInvokeContext::new() out of loop inside MessageProcessor::process_message().

* Moves MessageProcessor::verify_account_references() and MessageProcessor::verify() into InvokeContext::verify().
2021-09-27 08:28:45 +02:00
Jeff Washington (jwash) c6d189dac2
bg threads wait until next expected event (#20126) 2021-09-24 16:00:41 -05:00
Jeff Washington (jwash) 122206dbb1
AcctIdx: bg loop sleeps when all buckets flushed at current age (#20168) 2021-09-24 13:19:06 -05:00
Jeff Washington (jwash) 1d13594c1c
AcctIdx: store # threads in BucketHolder. Used later. (#20151) 2021-09-23 19:56:44 -05:00
Jeff Washington (jwash) b3bb079d9f
AcctIdx: condense upsert to always use read lock (#20148) 2021-09-23 19:19:27 -05:00
Brooks Prumo 1347b507da
Handle duplicate AppendVec IDs (#20096)
When reconstructing the AccountsDb, if the storages came from full and
incremental snapshots generated on different nodes, it's possible that
the AppendVec IDs could overlap/have duplicates, which would cause the
reconstruction to fail.

This commit handles this issue by unconditionally remapping the
AppendVec ID for every AppendVec.

Fixes #17088
2021-09-23 17:34:32 -05:00
Jeff Washington (jwash) 2ae1e80013
AcctIdx: items needs to load from disk (#20138) 2021-09-23 17:33:54 -05:00
Jeff Washington (jwash) 334d38d200
AcctIdx: generation per thread starts at a random bucket (#20122) 2021-09-23 19:59:37 +00:00
Jeff Washington (jwash) 254d9c8903
range takes a ref (#20140) 2021-09-23 13:57:56 -05:00
Jeff Washington (jwash) 093dd68214
AcctIdx: stats for how much time bg threads wait (#20127) 2021-09-23 18:37:14 +00:00
Jeff Washington (jwash) fe28f17718
AcctIdx: startup causes us to flush remove everything asap (#20121) 2021-09-23 12:17:52 -05:00
Jeff Washington (jwash) 5dae615208
AcctIdx: set age of cache accesses to future (#20125) 2021-09-23 11:43:48 -05:00
Jeff Washington (jwash) fb55e9a9b6
AcctIdx: when stop_flush is disabled, prepare to age (#20113) 2021-09-23 11:42:32 -05:00
Jeff Washington (jwash) 5bbb0da7b8
AcctIdx: stats for in-mem (#20114) 2021-09-23 08:14:53 -05:00
Jeff Washington (jwash) cb9d93525b
AcctIdx: metrics for loading from disk (#20124) 2021-09-23 08:14:08 -05:00
Jeff Washington (jwash) 4d3e32803f
AcctIdx: PreAllocatedAccountMapEntry (#20117) 2021-09-23 08:11:32 -05:00
Trent Nelson 7b365c564f
runtime: remove inactive delegation from stakes cache 2021-09-22 20:50:58 -06:00
Jeff Washington (jwash) 476124de50
AcctIdx: report disk bucket stats (#20120) 2021-09-22 18:17:20 -05:00
Jeff Washington (jwash) f7eebadbed
AcctIdx: remove old items from cache (#20076) 2021-09-22 21:37:15 +00:00
Jeff Washington (jwash) 0eb0d7f73b
AcctIdx: consolidate lock_and_update_slot_list (#20090) 2021-09-22 18:40:19 +00:00
Jeff Washington (jwash) 4fb77183ef
AcctIdx: move background() to bucket holder (#20085) 2021-09-22 17:40:30 +00:00
Jeff Washington (jwash) 681e8728a1
get rid of logs left in test (#20116) 2021-09-22 12:39:48 -05:00
Jeff Washington (jwash) bbd7bc2985
AcctIdx: Fix bad test (#20115) 2021-09-22 12:39:31 -05:00
Jeff Washington (jwash) 1eeddf2e23
orderings on AcctIdx dirty (#20087) 2021-09-22 11:24:16 -05:00
Jeff Washington (jwash) 681d6d121e
AcctIdx: 2 upserts use disk (#20072) 2021-09-22 11:14:08 -05:00
Jeff Washington (jwash) deab944b30
AcctIdx: flush inserts to disk (#20074) 2021-09-22 09:55:40 -05:00
Jeff Washington (jwash) 804ed825f7
AcctIdx: rework timing of bg loop (#20077) 2021-09-22 09:39:49 -05:00
Jeff Washington (jwash) a6aef76f52
AcctIdx: put_range_in_cache loads from disk (#20073) 2021-09-22 09:37:34 -05:00
Jeff Washington (jwash) c944bfb1d9
AcctIdx tests use disk buckets (#20070) 2021-09-22 08:55:58 -05:00
Jeff Washington (jwash) 9e9417b474
AcctIdx: initialze stats with # bins (#20088) 2021-09-22 08:55:07 -05:00
Jeff Washington (jwash) b918c2f4cd
AcctIdx: remove_if_slot_list_empty uses disk backing (#20069) 2021-09-22 08:48:44 -05:00
Jeff Washington (jwash) 6c518102dd
AcctIdx: fix some missing dirty calls (#20089) 2021-09-22 08:45:08 -05:00
dependabot[bot] 1b58166e9d
chore: bump memmap2 from 0.4.0 to 0.5.0 (#20057)
* chore: bump memmap2 from 0.4.0 to 0.5.0

Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/RazrFalcon/memmap2-rs/releases)
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-21 17:20:49 -06:00
Jeff Washington (jwash) e6795e6fdd
AcctIdx: get can load from disk (#20067) 2021-09-21 12:40:07 -05:00
Jeff Washington (jwash) 7f3d445af5
AcctIdx: Orderings and cleanup (#20046) 2021-09-21 10:52:39 -05:00
Jeff Washington (jwash) bbe151e594
AcctIdx: only flush if disk index being used (#20045) 2021-09-21 08:42:13 -05:00
Jeff Washington (jwash) 38ea3aa31a
AcctIdx: keys impl when items are on disk (#20039) 2021-09-21 08:41:39 -05:00
Jeff Washington (jwash) fb71e80962
AcctIdx: accessors for age (#20023) 2021-09-21 08:41:17 -05:00
Jeff Washington (jwash) c3679ab9bd
AcctIdx: first pass at aging buckets (#20008) 2021-09-21 08:40:55 -05:00
Alexander Meißner b507715d44
CPI without Account Refs (#20034)
* Removes search for accounts and unsafe lifetime transmute in InvokeContext::push().

* Replaces accounts by account_indices in verify_and_update() and process_cross_program_instruction().

Co-authored-by: Justin Starry <justin.m.starry@gmail.com>
2021-09-21 14:41:02 +02:00
Jeff Washington (jwash) 94330de843
AcctIdx: delete from disk index (#20035) 2021-09-20 16:19:21 -05:00
Jeff Washington (jwash) 6399a87485
AcctIdx: report age in stats (#20025) 2021-09-20 14:30:12 -05:00
Jeff Washington (jwash) e9ee90a121
AcctIdx: age adds time component (#20024) 2021-09-20 14:29:34 -05:00
Jeff Washington (jwash) d842705622
AcctIdx: when bg processing in multiple threads, avoid bin collisions (#20019) 2021-09-20 11:50:22 -05:00
Jeff Washington (jwash) e6934e7247
AcctIdx: consolidate to correct CondVar (#20017) 2021-09-20 09:58:20 -05:00
Jeff Washington (jwash) 55d9ff9899
WaitableCondVar supports notify_one (#20013) 2021-09-20 09:57:57 -05:00
Jeff Washington (jwash) bc7983b13e
AcctIdx: metric for active bg threads (#20026) 2021-09-20 09:44:40 -05:00
Jeff Washington (jwash) 2a42f8a06e
AcctIdx: disk index now holds a bucket map (#19995) 2021-09-20 08:40:10 -05:00
Jeff Washington (jwash) af5b0d42a8
AcctIdx: initial index items have future age to flush (#20010) 2021-09-19 20:22:09 -05:00
Jeff Washington (jwash) c1d181add5
add --accounts-index-memory-limit-mb (#19269) 2021-09-19 18:00:15 -05:00
Jeff Washington (jwash) f579f73700
AcctIdx: introduce age per in-mem bucket (#19982) 2021-09-18 22:09:37 -05:00
Jeff Washington (jwash) 742155c214
call set_startup and add metrics on generate_index (#20006) 2021-09-18 22:08:58 -05:00
Jeff Washington (jwash) db40d06a39
AcctIdx: introduce startup to bucket map (#20004) 2021-09-18 12:55:57 -05:00
Jeff Washington (jwash) bed0049a51
AcctIdx: add age per item in in_mem acct idx (#19981) 2021-09-18 09:56:26 -05:00
Jeff Washington (jwash) 8df8f4396d
AcctIdx: start calling flush on in mem cache buckets (#19966)
* AcctIdx: start calling flush on in mem cache buckets

* fix orderings
2021-09-18 09:56:08 -05:00
Jeff Washington (jwash) 4089f8b06b
AcctIdx: with multiple bg threads, only log stats once per time interval (#19984) 2021-09-18 09:55:20 -05:00
Jeff Washington (jwash) 4e038e94fd
add 'drives' to AccountsIndexConfig (#19989) 2021-09-18 09:54:57 -05:00
Jeff Washington (jwash) f500c99a6d
pass AcctIdxConfig to BucketMapHolder (#19997) 2021-09-18 09:54:00 -05:00
Jeff Washington (jwash) bd86f41e18
acct idx tests use 2 bins instead of 8k (#19996) 2021-09-18 09:53:44 -05:00
Jeff Washington (jwash) 4dc2f08198
AcctIdx: hold_range_in_memory (#19955) 2021-09-17 22:19:29 +00:00
Jeff Washington (jwash) 24b136a993
AcctIdx: configurable flush threads (#19983) 2021-09-17 22:02:43 +00:00
Jeff Washington (jwash) b2152be3b2
introduce bucket map (#19848)
* introduce bucket map

* rename BucketMap bits to num_buckets_pow2

* use u64::BITS

* Store the number of buckets in BucketMapConfig as a regular number

* remove redundant type aliases

* use Slot from sdk

* use Arc::clone() instead

* fixup erase drives

* rename num_buckets to max_buckets

* add doc to BucketMapConfig::new()

* add more documentation

* rename to DEFAULT_CAPACITY_POW2

* doc

* add more traits while we can

* rename capacity to capacity_pow2

* fix a naming for max_buckets_pow2

* remove unused/incorrect DataBucket::bytes

* rework benches a bit

* fixup bench docs

* rename create_bucket_capacity_pow2 to bucket_capacity_when_created_pow2

* rename BucketMapKeyValue to BucketItem

* rename to items_in_range

* remove values()

* remove addref and unref

* remove more addref and unref

* resurect addref and unref since tests use 'em for now

* rename to BucketStorage

* move stats in bucket_stats

* remove specializations (i don't think they are needed)

* move MaxSearch and RefCount into lib.rs

* move BucketItem to bucket_item.rs

* add doc

* keys no longer returns an option

* Revert "remove specializations (i don't think they are needed)"

This reverts commit b22f78e072cf0f7107851b08e58c2e3fead3f64d.

Co-authored-by: Brooks Prumo <brooks@solana.com>
2021-09-17 15:11:27 -05:00
Jeff Washington (jwash) cddb9da4f0
AcctIdx: central age functions/state (#19980) 2021-09-17 15:11:07 -05:00
Jeff Washington (jwash) 99f5684dc4
AcctIdx: handle future config defaults (#19967) 2021-09-17 13:12:06 -05:00
Jeff Washington (jwash) 9998e16df3
AcctIdx: prepare to spin up multiple bg threads (#19969) 2021-09-17 11:53:25 -05:00
Jack May 7b0bf64404
cleanup old features (#19956) 2021-09-17 09:46:49 -07:00
Jeff Washington (jwash) 37d6f5013f
AcctIdx: initialize acct idx by config (#19968) 2021-09-17 16:39:41 +00:00
dependabot[bot] b8106e2450
chore: bump memmap2 from 0.3.1 to 0.4.0 (#19974)
* chore: bump memmap2 from 0.3.1 to 0.4.0

Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.3.1 to 0.4.0.
- [Release notes](https://github.com/RazrFalcon/memmap2-rs/releases)
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.3.1...v0.4.0)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-09-17 10:00:12 -06:00
Jeff Washington (jwash) 376b20c3d2
AcctIdx: add some bucket flush helpers (#19965) 2021-09-17 10:41:30 -05:00
Jeff Washington (jwash) 4d8b3aa578
AcctsIdx: introduce dirty state per pubkey (#19960) 2021-09-16 21:36:23 -05:00
Jeff Washington (jwash) 58f25a8752
AcctIdx: move background() to AccountsIndexStorage (#19948) 2021-09-16 17:52:06 -05:00
Jeff Washington (jwash) 66e0fafc21
AccountsIndexStorage holds InMemAccountsIndex[] (#19947) 2021-09-16 21:12:22 +00:00
sakridge dc69cc1ae4
Only allow votes when root distance gets too high (#19917) 2021-09-16 15:12:26 +02:00
Jack May bd434d92e1
nit: misspelling (#19929) 2021-09-16 01:27:26 +00:00
dependabot[bot] fd60ef78eb
chore: bump flate2 from 1.0.21 to 1.0.22 (#19863)
* chore: bump flate2 from 1.0.21 to 1.0.22

Bumps [flate2](https://github.com/rust-lang/flate2-rs) from 1.0.21 to 1.0.22.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.0.21...1.0.22)

---
updated-dependencies:
- dependency-name: flate2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-15 14:30:30 -06:00
Jeff Washington (jwash) eddd583cd5
AcctIdx: pass bins to BucketMapHolder (#19906) 2021-09-15 18:07:53 +00:00
Jeff Washington (jwash) 99f2c746d1
AcctIdx: Put InMemAcctIndex behind arc (#19904) 2021-09-15 17:36:08 +00:00
Jeff Washington (jwash) b467e7fb3b
AcctIdx: remove auto Debug trait (#19905) 2021-09-15 09:54:16 -05:00
Brooks Prumo 8e3c420414
Add logging after taking a bank snapshot (#19891) 2021-09-15 07:39:21 -05:00
Jeff Washington (jwash) d3c80be7d5
AcctIdx: use stats() accessor (#19890) 2021-09-15 05:28:44 +00:00
Jeff Washington (jwash) 691bea8776
extract map to map() (#19883) 2021-09-14 23:08:17 -05:00
Jeff Washington (jwash) 229b378ed1
in-mem acct idx hash map uses rwlock (#19878) 2021-09-14 18:01:34 -05:00
Jeff Washington (jwash) fc647eed19
introduce trait IndexValue for accounts index (#19859) 2021-09-14 17:51:07 -05:00
Brooks Prumo 21bc7e94f5
Add logging of incremental snapshots to purge_old_snapshot_archives() (#19874) 2021-09-14 17:49:15 -05:00
Jeff Washington (jwash) 60276da1fd
rework acct idx entry to prepare for hashmap (#19875) 2021-09-14 15:27:23 -05:00
Jeff Washington (jwash) 2a795614d3
accounts index keys returns by value (#19858) 2021-09-14 09:38:59 -05:00
Jack May 00d7981f64
Fix native invoke writable privileges (#19750)
* Fix native invoke writable privileges

* build downstream spl bpf programs for tests
2021-09-13 22:57:37 -07:00
Jeff Washington (jwash) 910f241c3f
add <T> through accounts index (#19852) 2021-09-13 22:59:03 -05:00
Jeff Washington (jwash) b57e86abf2
cache account hash info (#19426)
* cache account hash info

* ledger_path -> accounts_hash_cache_path
2021-09-13 20:39:26 -05:00
Jeff Washington (jwash) dca49a614f
size and per-bin stats (#19842) 2021-09-13 19:26:49 -05:00
carllin 87a7f00926
Track reset bank in PohRecorder (#19810) 2021-09-13 16:55:35 -07:00
Brooks Prumo 3071c4caad
Set is_startup=true for exhaustively_free_unused_resource() (#19846) 2021-09-13 18:08:06 -05:00
Jeff Washington (jwash) e1a038e791
add stats inserts, deletes, updates_in_mem (#19807) 2021-09-13 15:59:01 -05:00
Brooks Prumo ef749a2506
Flush accounts cache before calling clean_accounts() (#19839) 2021-09-13 14:56:13 -05:00
Jeff Washington (jwash) c05226620f
metrics on accounts index entry (#19806) 2021-09-13 08:33:45 -05:00
Brooks Prumo 3f6eb96d6e
Call `Bank::update_accounts_hash()` before taking bank snapshot (#19765)
This is a bugfix.

When `load_frozen_forks()` called `snapshot_bank()`, the accounts hash
was not computed.  So then when a snapshot archive was eventually
created, its hash would be all 1s, which is clearly wrong.  The fix is
to update the bank's accounts hash before taking the bank snapshot.
2021-09-13 08:27:50 -05:00
Jeff Washington (jwash) d388c5c257
move update_key_if_exists to InMemAccountsIndex (#19803) 2021-09-13 08:27:21 -05:00
Jeff Washington (jwash) 3cea535fdb
InMemAccountsIndex::remove is no longer called (#19805) 2021-09-12 22:01:41 -05:00
Jeff Washington (jwash) b992c02708
in_mem_accounts_index filters by range (#19779) 2021-09-12 21:57:15 -05:00
Jeff Washington (jwash) 0263ffb2ed
move upsert and a handful of helpers to InMemAccountsIndex (#19799) 2021-09-12 21:54:09 -05:00
Jeff Washington (jwash) 3617d43e76
accounts index stats (#19797) 2021-09-12 22:14:59 +00:00
Jeff Washington (jwash) d9674f7ff0
consolidate another use of remove_if_slot_list_empty (#19804) 2021-09-12 19:05:44 +00:00
Brooks Prumo 62c8bcf565
Add default() to SnapshotConfig (#19776) 2021-09-12 13:44:27 -05:00
Jeff Washington (jwash) c9a3b8941f
move insert_new_entry_if_missing_with_lock into InMemAccountsIndex (#19800) 2021-09-12 13:39:29 -05:00
Jeff Washington (jwash) a295febf04
move remove_if_slot_list_empty to InMemAccountsIndex (#19801) 2021-09-12 10:26:04 -05:00
Jeff Washington (jwash) 361101bd31
rename generic V -> T. Matches intent better. V is now a larger type that contains T (#19798) 2021-09-12 10:25:09 -05:00
Giles Cope c9261e034e remove dev deps 2021-09-11 14:33:53 -07:00
Giles Cope c3a610374a unused deps 2021-09-11 14:33:53 -07:00
Jeff Washington (jwash) 91c3b18e1e
InMemAccountsIndex iter returns vec<owned> (#19778) 2021-09-11 20:33:04 +00:00
Jeff Washington (jwash) 7de2236284
accounts index get returns value (#19777) 2021-09-11 13:28:56 -05:00
Jeff Washington (jwash) ed7a6c6732
push use of AccountMapEntry one level deeper (#19775) 2021-09-11 02:09:51 +00:00
Jeff Washington (jwash) cba834808a
refactor generate_index zero lamports (#19769) 2021-09-10 23:46:08 +00:00
Jeff Washington (jwash) 595bba95b4
accounts index traits (#19768) 2021-09-10 18:32:10 -05:00
Jeff Washington (jwash) 9899cd359a
accounts index iterator uses copy of pubkey and account map entry (#19767) 2021-09-10 17:52:49 -05:00
Jeff Washington (jwash) 11b10439b4
factor out InMemAccountsIndex to prepare for disk index (#19773) 2021-09-10 17:52:25 -05:00