Commit Graph

129 Commits

Author SHA1 Message Date
Brooks Prumo 4d361af976
Add LastFullSnapshotSlot to SnapshotConfig (#19341) 2021-08-20 17:06:53 +00:00
Trent Nelson e0bc5fa690 validator: Trusted validators are now called known validators 2021-08-19 22:43:49 -06:00
Brooks Prumo f9986c66b8
Make SnapshotPackagerService aware of Incremental Snapshots (#19254)
Add a field to SnapshotPackage that is an enum for SnapshotType, so archive_snapshot_package() will do the right thing.

Fixes #19166
2021-08-17 13:01:59 -05:00
Brooks Prumo 176036aa58
Rename AccountsPacakge to SnapshotPackage and AccountsPackagePre to AccountsPackage (#19231)
Renaming these types to better communicate their usages, which will
further diverge as incremental snapshot support is added.

With the new names, AccountsPacakge now refers to the type between
AccountsBackgroundProcess and AccountsHashVerifier, and SnapshotPackage
refers to the type between AccountsHashVerifier and
SnapshotPackagerService.
2021-08-13 16:08:09 -05:00
Brooks Prumo ccfa82461b
Pass SnapshotConfig to AccountsHashVerifier (#19154)
AccountsHashVerifier will need access to both the full and incremental
snapshot archive interval slots config values, which is in the
SnapshotConfig.

Also, cleanup some `Option<>` params and their references.
2021-08-10 14:02:34 -05:00
Brooks Prumo fd937548a0
Move SnapshotArchiveInfo and friends into its own module (#19114) 2021-08-08 07:57:06 -05:00
Brooks Prumo 00890957ee
Add snapshot_utils::bank_from_latest_snapshot_archives() (#18983)
While reviewing PR #18565, as issue was brought up to refactor some code
around verifying the bank after rebuilding from snapshots.  A new
top-level function has been added to get the latest snapshot archives
and load the bank then verify.  Additionally, new tests have been
written and existing tests have been updated to use this new function.

Fixes #18973

While resolving the issue, it became clear there was some additional
low-hanging fruit this change enabled.  Specifically, the functions
`bank_to_xxx_snapshot_archive()` now return their respective
`SnapshotArchiveInfo`.  And on the flip side,
`bank_from_snapshot_archives()` now takes `SnapshotArchiveInfo`s instead
of separate paths and archive formats.  This bundling simplifies bank
rebuilding.
2021-08-06 20:16:06 -05:00
behzad nouri d2d5f36a3c
adds validator flag to allow private ip addresses (#18850) 2021-07-23 15:25:03 +00:00
Brooks Prumo d1debcd971
Add incremental snapshot utils (#18504)
This commit adds high-level functions for creating and loading-from
incremental snapshots, plus all low-level functions required to perform
those tasks.  This commit **does not** add taking incremental snapshots
as part of a running validator, nor starting up a node with an
incremental snapshot; just laying ground work.

Additionally, `snapshot_utils` and `serde_snapshot` have been
refactored to use a common code paths for the different snapshots.

Also of note, some renaming has happened:
  1. Snapshots are now either `full_` or `incremental_` throughout the
     codebase.  If not specified, the code applies to both.
  2. Bank snapshots now are called "bank snapshots"
     (before they were called "slot snapshots", "bank snapshots", or
      just "snapshots").  The one exception is within `Bank`, where they
     are still just "snapshots", because they are already "bank
     snapshots".
  3. Snapshot archives now have `_archive` in the code.  This
     should clear up an ambiguity between bank snapshots and snapshot
     archives.
2021-07-22 14:40:37 -05:00
Brooks Prumo 89a3e4f91e
Move SnapshotConfig into its own module (#18331)
Also move ArchiveFormat to snapshot_utils, and do not
reexport SnapshotVersion.
2021-07-01 08:55:26 -05:00
Alexander Meißner 6514096a67 chore: cargo +nightly clippy --fix -Z unstable-options 2021-06-18 10:42:46 -07:00
Tyera Eulberg 9a5330b7eb
Move gossip modules into solana-gossip crate (#17352)
* Move gossip modules to solana-gossip

* Update Protocol abi digest due to move

* Move gossip benches and hook up CI

* Remove unneeded Result entries

* Single use statements
2021-05-26 09:15:46 -06:00
Jeff Washington (jwash) 4aa753ff01
rename threads: 15 char limit (#16625) 2021-04-19 12:16:58 -05:00
Jeff Washington (jwash) fabecdc86c
use thread pool for non-index hash calculations (#15149) 2021-02-05 19:48:55 +00:00
Jeff Washington (jwash) 600ff0d915
calculate hash from store instead of index (#15034)
* calculate hash from store instead of index

* restore update hash in abs
2021-02-04 09:00:33 -06:00
Michael Vines a95675a7ce
Avoid tmp snapshot backlog in SnapshotPackagerService under high load (#14516) 2021-01-11 10:21:15 -08:00
Michael Vines 7be6770808 Rename CompressionType to ArchiveFormat 2021-01-09 09:07:49 -08:00
Michael Vines 141e6706e6 Rename AccountsPackage::root to AccountsPackage::slot 2021-01-09 09:07:49 -08:00
sakridge a5db6399ad
Fix test_max_hashes (#14189) 2020-12-17 15:12:18 -08:00
sakridge 1f1eb9f26e
Add separate push queue to reduce push lock contention (#12713) 2020-10-13 18:10:25 -07:00
Kristofer Peterson 6d81eede93
Add CLI options and runtime support for selection of output snapshot version. (#10536) 2020-06-19 14:38:37 +09:00
Greg Fitzgerald 6ee222363e
Move BankForks to solana_runtime (#10637)
* Move BankForks to solana_runtime

* Update imports
2020-06-17 15:27:03 +00:00
Kristofer Peterson 58ef02f02b
9951 clippy errors in the test suite (#10030)
automerge
2020-05-15 09:35:43 -07:00
carllin bab3502260
Push down cluster_info lock (#9594)
* Push down cluster_info lock

* Rework budget decrement

Co-authored-by: Carl <carl@solana.com>
2020-04-21 12:54:45 -07:00
sakridge 66abe45ea1
Decouple accounts hash calculation from snapshot hash (#9507) 2020-04-16 15:12:20 -07:00
sakridge 9493de4443
Add snapshot compression option (#9276) 2020-04-03 13:13:49 -07:00
sakridge a74a64084d
Fix error with account hash list getting too big for gossip (#9197)
Nodes will stop publishing hashes on gossip and also
slow memory leak.
2020-03-31 21:39:48 -07:00
sakridge c51049a59b
Add counter for accounts hash verification. (#8928) 2020-03-18 08:39:09 -07:00
sakridge dc347dd3d7
Add Accounts hash consistency halting (#8772)
* Accounts hash consistency halting

* Add option to inject account hash faults for testing.

Enable option in local cluster test to see that node halts.
2020-03-16 08:37:31 -07:00