Commit Graph

2018 Commits

Author SHA1 Message Date
Jeff Washington (jwash) c3679ab9bd
AcctIdx: first pass at aging buckets (#20008) 2021-09-21 08:40:55 -05:00
Alexander Meißner b507715d44
CPI without Account Refs (#20034)
* Removes search for accounts and unsafe lifetime transmute in InvokeContext::push().

* Replaces accounts by account_indices in verify_and_update() and process_cross_program_instruction().

Co-authored-by: Justin Starry <justin.m.starry@gmail.com>
2021-09-21 14:41:02 +02:00
Jeff Washington (jwash) 94330de843
AcctIdx: delete from disk index (#20035) 2021-09-20 16:19:21 -05:00
Jeff Washington (jwash) 6399a87485
AcctIdx: report age in stats (#20025) 2021-09-20 14:30:12 -05:00
Jeff Washington (jwash) e9ee90a121
AcctIdx: age adds time component (#20024) 2021-09-20 14:29:34 -05:00
Jeff Washington (jwash) d842705622
AcctIdx: when bg processing in multiple threads, avoid bin collisions (#20019) 2021-09-20 11:50:22 -05:00
Jeff Washington (jwash) e6934e7247
AcctIdx: consolidate to correct CondVar (#20017) 2021-09-20 09:58:20 -05:00
Jeff Washington (jwash) 55d9ff9899
WaitableCondVar supports notify_one (#20013) 2021-09-20 09:57:57 -05:00
Jeff Washington (jwash) bc7983b13e
AcctIdx: metric for active bg threads (#20026) 2021-09-20 09:44:40 -05:00
Jeff Washington (jwash) 2a42f8a06e
AcctIdx: disk index now holds a bucket map (#19995) 2021-09-20 08:40:10 -05:00
Jeff Washington (jwash) af5b0d42a8
AcctIdx: initial index items have future age to flush (#20010) 2021-09-19 20:22:09 -05:00
Jeff Washington (jwash) c1d181add5
add --accounts-index-memory-limit-mb (#19269) 2021-09-19 18:00:15 -05:00
Jeff Washington (jwash) f579f73700
AcctIdx: introduce age per in-mem bucket (#19982) 2021-09-18 22:09:37 -05:00
Jeff Washington (jwash) 742155c214
call set_startup and add metrics on generate_index (#20006) 2021-09-18 22:08:58 -05:00
Jeff Washington (jwash) db40d06a39
AcctIdx: introduce startup to bucket map (#20004) 2021-09-18 12:55:57 -05:00
Jeff Washington (jwash) bed0049a51
AcctIdx: add age per item in in_mem acct idx (#19981) 2021-09-18 09:56:26 -05:00
Jeff Washington (jwash) 8df8f4396d
AcctIdx: start calling flush on in mem cache buckets (#19966)
* AcctIdx: start calling flush on in mem cache buckets

* fix orderings
2021-09-18 09:56:08 -05:00
Jeff Washington (jwash) 4089f8b06b
AcctIdx: with multiple bg threads, only log stats once per time interval (#19984) 2021-09-18 09:55:20 -05:00
Jeff Washington (jwash) 4e038e94fd
add 'drives' to AccountsIndexConfig (#19989) 2021-09-18 09:54:57 -05:00
Jeff Washington (jwash) f500c99a6d
pass AcctIdxConfig to BucketMapHolder (#19997) 2021-09-18 09:54:00 -05:00
Jeff Washington (jwash) bd86f41e18
acct idx tests use 2 bins instead of 8k (#19996) 2021-09-18 09:53:44 -05:00
Jeff Washington (jwash) 4dc2f08198
AcctIdx: hold_range_in_memory (#19955) 2021-09-17 22:19:29 +00:00
Jeff Washington (jwash) 24b136a993
AcctIdx: configurable flush threads (#19983) 2021-09-17 22:02:43 +00:00
Jeff Washington (jwash) b2152be3b2
introduce bucket map (#19848)
* introduce bucket map

* rename BucketMap bits to num_buckets_pow2

* use u64::BITS

* Store the number of buckets in BucketMapConfig as a regular number

* remove redundant type aliases

* use Slot from sdk

* use Arc::clone() instead

* fixup erase drives

* rename num_buckets to max_buckets

* add doc to BucketMapConfig::new()

* add more documentation

* rename to DEFAULT_CAPACITY_POW2

* doc

* add more traits while we can

* rename capacity to capacity_pow2

* fix a naming for max_buckets_pow2

* remove unused/incorrect DataBucket::bytes

* rework benches a bit

* fixup bench docs

* rename create_bucket_capacity_pow2 to bucket_capacity_when_created_pow2

* rename BucketMapKeyValue to BucketItem

* rename to items_in_range

* remove values()

* remove addref and unref

* remove more addref and unref

* resurect addref and unref since tests use 'em for now

* rename to BucketStorage

* move stats in bucket_stats

* remove specializations (i don't think they are needed)

* move MaxSearch and RefCount into lib.rs

* move BucketItem to bucket_item.rs

* add doc

* keys no longer returns an option

* Revert "remove specializations (i don't think they are needed)"

This reverts commit b22f78e072cf0f7107851b08e58c2e3fead3f64d.

Co-authored-by: Brooks Prumo <brooks@solana.com>
2021-09-17 15:11:27 -05:00
Jeff Washington (jwash) cddb9da4f0
AcctIdx: central age functions/state (#19980) 2021-09-17 15:11:07 -05:00
Jeff Washington (jwash) 99f5684dc4
AcctIdx: handle future config defaults (#19967) 2021-09-17 13:12:06 -05:00
Jeff Washington (jwash) 9998e16df3
AcctIdx: prepare to spin up multiple bg threads (#19969) 2021-09-17 11:53:25 -05:00
Jack May 7b0bf64404
cleanup old features (#19956) 2021-09-17 09:46:49 -07:00
Jeff Washington (jwash) 37d6f5013f
AcctIdx: initialize acct idx by config (#19968) 2021-09-17 16:39:41 +00:00
dependabot[bot] b8106e2450
chore: bump memmap2 from 0.3.1 to 0.4.0 (#19974)
* chore: bump memmap2 from 0.3.1 to 0.4.0

Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.3.1 to 0.4.0.
- [Release notes](https://github.com/RazrFalcon/memmap2-rs/releases)
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.3.1...v0.4.0)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-09-17 10:00:12 -06:00
Jeff Washington (jwash) 376b20c3d2
AcctIdx: add some bucket flush helpers (#19965) 2021-09-17 10:41:30 -05:00
Jeff Washington (jwash) 4d8b3aa578
AcctsIdx: introduce dirty state per pubkey (#19960) 2021-09-16 21:36:23 -05:00
Jeff Washington (jwash) 58f25a8752
AcctIdx: move background() to AccountsIndexStorage (#19948) 2021-09-16 17:52:06 -05:00
Jeff Washington (jwash) 66e0fafc21
AccountsIndexStorage holds InMemAccountsIndex[] (#19947) 2021-09-16 21:12:22 +00:00
sakridge dc69cc1ae4
Only allow votes when root distance gets too high (#19917) 2021-09-16 15:12:26 +02:00
Jack May bd434d92e1
nit: misspelling (#19929) 2021-09-16 01:27:26 +00:00
dependabot[bot] fd60ef78eb
chore: bump flate2 from 1.0.21 to 1.0.22 (#19863)
* chore: bump flate2 from 1.0.21 to 1.0.22

Bumps [flate2](https://github.com/rust-lang/flate2-rs) from 1.0.21 to 1.0.22.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.0.21...1.0.22)

---
updated-dependencies:
- dependency-name: flate2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-15 14:30:30 -06:00
Jeff Washington (jwash) eddd583cd5
AcctIdx: pass bins to BucketMapHolder (#19906) 2021-09-15 18:07:53 +00:00
Jeff Washington (jwash) 99f2c746d1
AcctIdx: Put InMemAcctIndex behind arc (#19904) 2021-09-15 17:36:08 +00:00
Jeff Washington (jwash) b467e7fb3b
AcctIdx: remove auto Debug trait (#19905) 2021-09-15 09:54:16 -05:00
Brooks Prumo 8e3c420414
Add logging after taking a bank snapshot (#19891) 2021-09-15 07:39:21 -05:00
Jeff Washington (jwash) d3c80be7d5
AcctIdx: use stats() accessor (#19890) 2021-09-15 05:28:44 +00:00
Jeff Washington (jwash) 691bea8776
extract map to map() (#19883) 2021-09-14 23:08:17 -05:00
Jeff Washington (jwash) 229b378ed1
in-mem acct idx hash map uses rwlock (#19878) 2021-09-14 18:01:34 -05:00
Jeff Washington (jwash) fc647eed19
introduce trait IndexValue for accounts index (#19859) 2021-09-14 17:51:07 -05:00
Brooks Prumo 21bc7e94f5
Add logging of incremental snapshots to purge_old_snapshot_archives() (#19874) 2021-09-14 17:49:15 -05:00
Jeff Washington (jwash) 60276da1fd
rework acct idx entry to prepare for hashmap (#19875) 2021-09-14 15:27:23 -05:00
Jeff Washington (jwash) 2a795614d3
accounts index keys returns by value (#19858) 2021-09-14 09:38:59 -05:00
Jack May 00d7981f64
Fix native invoke writable privileges (#19750)
* Fix native invoke writable privileges

* build downstream spl bpf programs for tests
2021-09-13 22:57:37 -07:00
Jeff Washington (jwash) 910f241c3f
add <T> through accounts index (#19852) 2021-09-13 22:59:03 -05:00
Jeff Washington (jwash) b57e86abf2
cache account hash info (#19426)
* cache account hash info

* ledger_path -> accounts_hash_cache_path
2021-09-13 20:39:26 -05:00
Jeff Washington (jwash) dca49a614f
size and per-bin stats (#19842) 2021-09-13 19:26:49 -05:00
carllin 87a7f00926
Track reset bank in PohRecorder (#19810) 2021-09-13 16:55:35 -07:00
Brooks Prumo 3071c4caad
Set is_startup=true for exhaustively_free_unused_resource() (#19846) 2021-09-13 18:08:06 -05:00
Jeff Washington (jwash) e1a038e791
add stats inserts, deletes, updates_in_mem (#19807) 2021-09-13 15:59:01 -05:00
Brooks Prumo ef749a2506
Flush accounts cache before calling clean_accounts() (#19839) 2021-09-13 14:56:13 -05:00
Jeff Washington (jwash) c05226620f
metrics on accounts index entry (#19806) 2021-09-13 08:33:45 -05:00
Brooks Prumo 3f6eb96d6e
Call `Bank::update_accounts_hash()` before taking bank snapshot (#19765)
This is a bugfix.

When `load_frozen_forks()` called `snapshot_bank()`, the accounts hash
was not computed.  So then when a snapshot archive was eventually
created, its hash would be all 1s, which is clearly wrong.  The fix is
to update the bank's accounts hash before taking the bank snapshot.
2021-09-13 08:27:50 -05:00
Jeff Washington (jwash) d388c5c257
move update_key_if_exists to InMemAccountsIndex (#19803) 2021-09-13 08:27:21 -05:00
Jeff Washington (jwash) 3cea535fdb
InMemAccountsIndex::remove is no longer called (#19805) 2021-09-12 22:01:41 -05:00
Jeff Washington (jwash) b992c02708
in_mem_accounts_index filters by range (#19779) 2021-09-12 21:57:15 -05:00
Jeff Washington (jwash) 0263ffb2ed
move upsert and a handful of helpers to InMemAccountsIndex (#19799) 2021-09-12 21:54:09 -05:00
Jeff Washington (jwash) 3617d43e76
accounts index stats (#19797) 2021-09-12 22:14:59 +00:00
Jeff Washington (jwash) d9674f7ff0
consolidate another use of remove_if_slot_list_empty (#19804) 2021-09-12 19:05:44 +00:00
Brooks Prumo 62c8bcf565
Add default() to SnapshotConfig (#19776) 2021-09-12 13:44:27 -05:00
Jeff Washington (jwash) c9a3b8941f
move insert_new_entry_if_missing_with_lock into InMemAccountsIndex (#19800) 2021-09-12 13:39:29 -05:00
Jeff Washington (jwash) a295febf04
move remove_if_slot_list_empty to InMemAccountsIndex (#19801) 2021-09-12 10:26:04 -05:00
Jeff Washington (jwash) 361101bd31
rename generic V -> T. Matches intent better. V is now a larger type that contains T (#19798) 2021-09-12 10:25:09 -05:00
Giles Cope c9261e034e remove dev deps 2021-09-11 14:33:53 -07:00
Giles Cope c3a610374a unused deps 2021-09-11 14:33:53 -07:00
Jeff Washington (jwash) 91c3b18e1e
InMemAccountsIndex iter returns vec<owned> (#19778) 2021-09-11 20:33:04 +00:00
Jeff Washington (jwash) 7de2236284
accounts index get returns value (#19777) 2021-09-11 13:28:56 -05:00
Jeff Washington (jwash) ed7a6c6732
push use of AccountMapEntry one level deeper (#19775) 2021-09-11 02:09:51 +00:00
Jeff Washington (jwash) cba834808a
refactor generate_index zero lamports (#19769) 2021-09-10 23:46:08 +00:00
Jeff Washington (jwash) 595bba95b4
accounts index traits (#19768) 2021-09-10 18:32:10 -05:00
Jeff Washington (jwash) 9899cd359a
accounts index iterator uses copy of pubkey and account map entry (#19767) 2021-09-10 17:52:49 -05:00
Jeff Washington (jwash) 11b10439b4
factor out InMemAccountsIndex to prepare for disk index (#19773) 2021-09-10 17:52:25 -05:00
Brooks Prumo 7aa5f6b833
Add CLI args for incremental snapshots (#19694)
Add `--incremental-snapshots` flag to enable incremental snapshots.
This will allow setting `--full-snapshot-interval-slots` and
`--incremental-snapshot-interval-slots`.

Also added `--maximum-incremental-snapshots-to-retain`.

Co-authored-by: Michael Vines <mvines@gmail.com>
2021-09-10 15:59:26 -05:00
Jeff Washington (jwash) 2de2c6ddd7
log accounts loaded for shrink stats (#19747) 2021-09-10 12:51:58 -05:00
Sean Young 098585234d Add return data implementation
This consists of:
 - syscalls
 - passing return data from invoked to invoker
 - printing to stable log
 - rust and C SDK changes
2021-09-10 14:25:54 +01:00
Alexander Meißner 88c1b8f047
Index loaders / executable accounts (#19469)
* Appends loaders / executable_accounts to accounts in transaction loading.

* Adds indices to loaders / executable_accounts.

* Moves MessageProcessor::create_keyed_accounts() into InvokeContext::push().

* Removes "executable_accounts",
now referenced by transaction wide index into "accounts".

* Removes create_pre_accounts() from InstructionProcessor,
as it is already in MessageProcessor.

* Collect program account indices directly in load_executable_accounts().
2021-09-10 08:36:21 +02:00
Jeff Washington (jwash) df5befdaca
metrics on rent partitions (#19728) 2021-09-09 10:50:35 -05:00
Jeff Washington (jwash) 8489ee7df9
Revert "collect rent in parallel for partitions (#19478)" (#19723)
This reverts commit 1560d5e3c3.
2021-09-08 21:27:08 -05:00
Jeff Washington (jwash) 97418bede3
introduce BinnedHashData type to improve some code readability (#19722) 2021-09-08 19:25:52 -05:00
Jeff Washington (jwash) 1cf7ae51f8
add comment (#19705) 2021-09-08 19:25:20 -05:00
Jeff Washington (jwash) 001ed66b4d
minor constant cleanup (#19721) 2021-09-08 18:26:10 -05:00
Tyera Eulberg 38bbb77989
Return error if Transaction contains writable executable or ProgramData accounts (#19629)
* Return error if Transaction locks an executable as writable

* Return error if a ProgramData account is writable but the upgradable loader isn't present

* Remove unreachable clause

* Fixup bpf tests

* Review comments

* Add new TransactionError

* Disallow writes to any upgradeable-loader account when loader not present; remove is_upgradeable_loader_present exception for all other executables
2021-09-08 15:21:52 -06:00
sakridge 3a8c678f62
Remove some copying (#19691) 2021-09-08 18:32:38 +02:00
Jeff Washington (jwash) 456bf15012
AccountsIndexConfig -> AccountsDbConfig (#19687) 2021-09-08 04:30:38 +00:00
Jeff Washington (jwash) d3f938f0cf
Remove Copy from AccountsIndexConfig. Not all types will support it (#19686) 2021-09-07 20:09:40 -05:00
Brooks Prumo 4a5f83d3a7
Add fn to check when to take snapshots (#19682) 2021-09-07 23:26:35 +00:00
Brooks Prumo a0552e5b46
Make startup aware of Incremental Snapshots (#19600) 2021-09-07 20:43:43 +00:00
Jeff Washington (jwash) 982454a455
refactor do_shrink_slot_stores (#19560) 2021-09-07 15:10:49 -05:00
Jeff Washington (jwash) 1560d5e3c3
collect rent in parallel for partitions (#19478)
* collect rent in parallel for partitions

* combine use

* use fg accounts db thread pool
2021-09-07 11:51:59 -05:00
Brooks Prumo 9d9482b9d8
Plumb `maximum_incremental_snapshot_archives_to_retain` (#19640) 2021-09-06 18:01:56 -05:00
Sean Young 8b9e472a6c feat: add ed25519 signature verify program
Solang requires a method for verify ed25519 signatures. Add a new
builtin program at address Ed25519SigVerify111111111111111111111111111
which takes any number of ed25519 signature, public key, and message.
If any of the signatures fails to verify, an error is returned.

The changes for the web3.js package will go into another commit, since
the tests test against a released solana node. Adding web3.js ed25519
testing will break CI.
2021-09-05 18:59:37 +01:00
Sean Young d461a9ac10 verify_precompiles needs FeatureSet
Rather than pass in individual features, pass in the entire feature set
so that we can add the ed25519 program feature in a later commit.
2021-09-05 18:59:37 +01:00
Brooks Prumo 333e5a9446
Make purge_old_snapshot_archives aware of maximum_incremental_snapshot_archives_to_retain (#19615) 2021-09-04 07:37:29 -05:00
Tyera Eulberg decec3cd8b
Demote write locks on transaction program ids (#19593)
* Add feature

* Demote write lock on program ids

* Fixup bpf tests

* Update MappedMessage::is_writable

* Comma nit

* Review comments
2021-09-04 03:05:30 +00:00
Jeff Washington (jwash) 7578db7ee3
add lowest_pubkey_from_bin (#19617) 2021-09-04 02:20:55 +00:00
Jeff Washington (jwash) b3fae0a01a
move constants in functions to global space (#19622) 2021-09-03 19:46:42 -05:00
Jeff Washington (jwash) e6055010eb
accounts index iterator takes &Range (#19618) 2021-09-03 18:00:49 -05:00
Jon Cinque 2c3bdedea3
stake: Advance `credits_observed` on activation epoch (#19309)
* stake: Advance `credits_observed` on activation epoch

* Add test for merging stakes just after activation
2021-09-03 23:20:56 +02:00
Brooks Prumo 5e25ee5ebe
Add maximum_incremental_snapshot_archives_to_retain to SnapshotConfig (#19612) 2021-09-03 20:21:32 +00:00
Brooks Prumo 7ab0aec61f
Rename maximum_full_snapshot_archives_to_retain (#19610)
To prepare for adding maximum_incremental_snapshot_archives_to_retain,
rename the current field in SnapshotConfig.
2021-09-03 11:28:10 -05:00
Jeff Washington (jwash) df040c05da
get rid of unneeded mut and refactor addref (#19594) 2021-09-03 10:45:37 -05:00
Jack May 170927b7c4
nit: Move builtins types to builtins file (#19597) 2021-09-02 21:29:11 -07:00
Jeff Washington (jwash) 3936c10aa0
rename var (#19588) 2021-09-03 00:55:35 +00:00
Jeff Washington (jwash) 9cc57fb076
remove file that accidentally got added (#19586) 2021-09-03 00:15:25 +00:00
Brooks Prumo e9374d32a3
Revert "Make startup aware of Incremental Snapshots (#19550)" (#19599)
This reverts commit d45ced0a5d.
2021-09-02 19:14:41 -05:00
Brooks Prumo d45ced0a5d
Make startup aware of Incremental Snapshots (#19550) 2021-09-02 19:05:15 -05:00
Jeff Washington (jwash) 57f51352f6
refactor AccountEntry addref/unref (#19583) 2021-09-02 23:25:27 +00:00
Jeff Washington (jwash) 682daf1117
cleanup of allocate_accounts_index (#19585) 2021-09-02 22:58:07 +00:00
Jeff Washington (jwash) 78585a992b
add IsCached trait bounds to AccountsIndex (#19584) 2021-09-02 22:45:35 +00:00
Jeff Washington (jwash) 2874f6b0a2
create remove_if_slot_list_empty to define accounts index behavior (#19581) 2021-09-02 16:54:37 -05:00
Jeff Washington (jwash) 0571962776
accounts index test uses # test bins (#19580) 2021-09-02 16:32:42 -05:00
Jeff Washington (jwash) 6ab8286e8d
add logging for a handful of tests (#19582) 2021-09-02 16:32:19 -05:00
Jeff Washington (jwash) f5388cfe71
tests which assert or panic in other threads now will report an error (#19532)
* tests which assert or panic in other threads now will report an error

* rename, add assert detecting time out
2021-09-02 11:26:57 -05:00
Jeff Washington (jwash) 7eb793a55e
add debugging to test (#19563)
* add debugging to test

* update cargo
2021-09-02 02:52:16 +00:00
Jeff Washington (jwash) ed2c071fe1
remove unused is_startup (#19561) 2021-09-01 18:39:00 -05:00
Lijun Wang 8378e8790f
Accountsdb replication installment 2 (#19325)
This is the 2nd installment for the AccountsDb replication.

Summary of Changes

The basic google protocol buffer protocol for replicating updated slots and accounts. tonic/tokio is used for transporting the messages.

The basic framework of the client and server for replicating slots and accounts -- the persisting of accounts in the replica-side will be done at the next PR -- right now -- the accounts are streamed to the replica-node and dumped. Replication for information about Bank is also not done in this PR -- to be addressed in the next PR to limit the change size.

Functionality used by both the client and server side are encapsulated in the replica-lib crate.

There is no impact to the existing validator by default.

Tests:

Observe the confirmed slots replicated to the replica-node.
Observe the accounts for the confirmed slot are received at the replica-node side.
2021-09-01 14:10:16 -07:00
Jeff Washington (jwash) a5f74d86c8
make test pass with 1 bin to remove noisy failure during testing (#19554) 2021-09-01 15:58:56 -05:00
Jeff Washington (jwash) dd9481c403
parallelize index_read in shrink (#19506) 2021-09-01 15:40:59 -05:00
Jeff Washington (jwash) 82a6bbe068
add Debug trait to range for accounts index (#19531) 2021-09-01 08:13:56 -05:00
Jeff Washington (jwash) 12dc8749cf
add Copy trait to AccountInfo for fast copies to mmapped file (#19524) 2021-09-01 08:13:08 -05:00
Tyera Eulberg 0088aefa24
Fix tests that make assumptions about tx fee rate (#19537) 2021-08-31 22:02:14 -06:00
Brooks Prumo 1d5a8ebc6a
Revert "Add LastFullSnapshotSlot to SnapshotConfig (#19341)" (#19529)
This reverts commit 4d361af976.
2021-08-31 22:03:19 -05:00
Jeff Washington (jwash) 2df96cd81e
remove type specifics to make funciton more flexible (#19536) 2021-08-31 20:03:42 -05:00
Brooks Prumo fe9ee9134a
Make background services aware of incremental snapshots (#19401)
AccountsBackgroundService now knows about incremental snapshots.  It is
now also in charge of deciding if an AccountsPackage is destined to be a
SnapshotPackage or not (or just used by AccountsHashVerifier).

!!! New behavior changes !!!

Taking snapshots (both bank and archive) **MUST** succeed.

This is required because of how the last full snapshot slot is
calculated, which is used by AccountsBackgroundService when calling
`clean_accounts()`.

File system calls are now unwrapped and will result in a crash. As Trent told me:

>Well I think if a snapshot fails due to some IO error, it's very likely that the operator is going to have to intervene before it works.  We should exit error in this case, otherwise the validator might happily spin for several more hours, never successfully writing a complete snapshot, before something else brings it down.  This would leave the validator's last local snapshot many more slots behind than it would be had we exited outright and potentially force the operator to abandon ledger continuity in favor of a quick catchup

Other errors will set the `exit` flag to `true`, and the node will gracefully shutdown.

Fixes #19167 
Fixes #19168
2021-08-31 18:33:27 -05:00
Brooks Prumo 718ab7c12e Revert "Add snapshot_runtime_info module (#18199)"
This reverts commit c1b9d40a64.
2021-08-31 17:50:18 -05:00
Jack May 5cf8d8795b
Feature cleanup (#19528) 2021-08-31 14:51:26 -07:00
Jeff Washington (jwash) e1939688c2
parallel shrink (#19507) 2021-08-31 11:57:34 -05:00
Jeff Washington (jwash) 0e94b8fd4e
remove redundant cancelled_shrink (#19522) 2021-08-31 11:51:48 -05:00
Jeff Washington (jwash) 9d3afba045
Debug shrink stats (#19505)
* add some shrink metrics

* renames
2021-08-31 09:04:56 -05:00
Lijun Wang 09458cc802
Shrink mmap leakage (#19373)
* Putting stores satisfying future shrink into the candidate list.

* Fixed unit tests for select_shrink_candidates

* Added metrics on stores readded back for next round

* Ehance the metrics on accountsdb to report the total bytes and alive bytes and alive ratio

* Enhance select_shrink_candidates metrics
2021-08-31 00:55:16 -07:00
Jeff Washington (jwash) a655b01700
some basic accounts index refactoring (#19510) 2021-08-30 18:40:10 -05:00
dependabot[bot] 6cfd44b811
chore: bump flate2 from 1.0.20 to 1.0.21 (#19500)
* chore: bump flate2 from 1.0.20 to 1.0.21

Bumps [flate2](https://github.com/rust-lang/flate2-rs) from 1.0.20 to 1.0.21.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.0.20...1.0.21)

---
updated-dependencies:
- dependency-name: flate2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-08-30 16:48:02 -06:00
dependabot[bot] 6a53ec28e0
chore: bump serde from 1.0.129 to 1.0.130 (#19497)
* chore: bump serde from 1.0.129 to 1.0.130

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.129 to 1.0.130.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.129...v1.0.130)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-30 18:19:34 +00:00
behzad nouri 8ad52fa095
implements copy-on-write for vote-accounts (#19362)
Bank::vote_accounts redundantly clones vote-accounts HashMap even though
an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/95c998a19/runtime/src/bank.rs#L5174-L5186

This commit implements copy-on-write semantics for vote-accounts by
wrapping the underlying HashMap in Arc<...>.
2021-08-30 15:54:01 +00:00
Justin Starry 2d7f036afd
Add solana-program-runtime crate (#19438) 2021-08-27 00:30:36 +00:00
Jeff Washington (jwash) 02b050e0f5
replace AccountsIndex btree with hashmap of 8k bins (#19212) 2021-08-26 18:12:43 -05:00
Jeff Washington (jwash) 98bc694606
hash calculation adds really old slots to dirty_stores (#19434) 2021-08-26 14:32:43 -05:00
Jeff Washington (jwash) 535de3b302
add test for test_clean_nonrooted (#19409) 2021-08-26 13:56:09 -05:00
dependabot[bot] e492638b7d
chore: bump libc from 0.2.100 to 0.2.101 (#19442)
* chore: bump libc from 0.2.100 to 0.2.101

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.100 to 0.2.101.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.100...0.2.101)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2021-08-26 12:38:34 -06:00
dependabot[bot] 46890ac197
chore: bump serde from 1.0.128 to 1.0.129 (#19395)
* chore: bump serde from 1.0.128 to 1.0.129

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.128 to 1.0.129.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.128...v1.0.129)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-24 12:32:30 -06:00
dependabot[bot] c0c95e88d8
chore: bump libc from 0.2.99 to 0.2.100 (#19371)
* chore: bump libc from 0.2.99 to 0.2.100

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.99 to 0.2.100.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.99...0.2.100)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-23 09:34:32 -06:00
dependabot[bot] aea9960c75
chore: bump serde from 1.0.127 to 1.0.128 (#19370)
* chore: bump serde from 1.0.127 to 1.0.128

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.127 to 1.0.128.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.127...v1.0.128)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2021-08-23 09:33:34 -06:00
Brooks Prumo 1a4bede918
Remove filter_snapshot_storages_for_incremental_snapshot() (#19349)
Now that we can get just the subset of snapshot storages we want, the extra
filtering afterwards is redundant.
2021-08-23 09:45:29 -05:00
Brooks Prumo 6d939811e9
Name snapshots consistently (#19346)
#### Problem

Snapshot names are overloaded, and there are multiple terms that mean the same thing. This is confusing. Here's a list of ones in the codebase that I've found:

```
- snapshot_dir
- snapshots_dir
- snapshot_path
- snapshot_output_dir
- snapshot_package_output_path
- snapshot_archives_dir
```

#### Summary of Changes

For all the ones that are about the directory where snapshot archives are stored, ensure they are `snapshot_archives_dir`. For the ones about the (bank) snapshots directory, set to `bank_snapshots_dir`.


Co-authored-by: Michael Vines <mvines@gmail.com>
2021-08-21 15:41:03 -05:00
Brooks Prumo 234461f779
Add base_slot to get_snapshot_storages() (#19348) 2021-08-20 16:23:43 -05:00