Jeff Washington (jwash)
334e1112a7
during untar, read ahead, causing decompression to occur in the bg ( #18205 )
...
* shared buffered decompressor for untar
* spell check in comment
2021-07-08 13:44:47 -05:00
dependabot[bot]
0426c2d28e
chore: bump bzip2 from 0.3.3 to 0.4.3 ( #18361 )
...
* chore: bump bzip2 from 0.3.3 to 0.4.3
Bumps [bzip2](https://github.com/alexcrichton/bzip2-rs ) from 0.3.3 to 0.4.3.
- [Release notes](https://github.com/alexcrichton/bzip2-rs/releases )
- [Commits](https://github.com/alexcrichton/bzip2-rs/commits )
---
updated-dependencies:
- dependency-name: bzip2
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Update to new api
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-07-01 13:10:52 -06:00
Brooks Prumo
45d54b1fc6
Add SnapshotArchiveInfo and refactor functions in snapshot_utils ( #18232 )
2021-07-01 12:20:56 -05:00
Brooks Prumo
89a3e4f91e
Move SnapshotConfig into its own module ( #18331 )
...
Also move ArchiveFormat to snapshot_utils, and do not
reexport SnapshotVersion.
2021-07-01 08:55:26 -05:00
Jeff Washington (jwash)
ce53b84cdc
refactor untar_snapshot_in to push parallelism deeper for further refactoring ( #18310 )
2021-06-29 18:26:15 -05:00
Jeff Washington (jwash)
31ec986ea1
untar in parallel ( #18184 )
...
* untar in parallel
* make enum for 'ignore' return value
2021-06-24 17:29:49 -05:00
Alexander Meißner
6514096a67
chore: cargo +nightly clippy --fix -Z unstable-options
2021-06-18 10:42:46 -07:00
Jeff Washington (jwash)
bb7413c9f9
refactor rebuild_bank_from_snapshots ( #17988 )
2021-06-16 11:01:52 -05:00
Jeff Washington (jwash)
dbd4dc04b0
ledger tool limit_load_slot_count_from_snapshot avoids assert failures ( #17974 )
2021-06-15 15:39:22 -05:00
Jeff Washington (jwash)
f558b9b6bf
verify bank hash on startup with ledger tool option ( #17939 )
2021-06-15 11:52:12 -05:00
Jeff Washington (jwash)
471b34132e
add metrics for startup ( #17913 )
...
* add metrics for startup
* roll timings up higher
* fix test
* fix duplicate
2021-06-14 17:46:49 -05:00
Jeff Washington (jwash)
d4cc975fe9
calculate_capitalization uses hash calculation ( #17443 )
...
* calculate_capitalization uses hash calculation
* feedback
* remove debugging code, clean up slot math
2021-06-14 08:53:07 -05:00
Lijun Wang
269d995832
Make account shrink configurable #17544 ( #17778 )
...
1. Added both options for measuring space usage using total accounts usage and for individual store shrink ratio using an enum. Validator CLI options: --accounts-shrink-optimize-total-space and --accounts-shrink-ratio
2. Added code for selecting candidates based on total usage in a separate function select_candidates_by_total_usage
3. Added unit tests for the new functions added
4. The default implementations is kept at 0.8 shrink ratio with --accounts-shrink-optimize-total-space set to true
Fixes #17544
2021-06-09 21:21:32 -07:00
Jeff Washington (jwash)
9388aaca15
use slots returned from get_snapshot_storages to sort ( #17638 )
...
* use slots returned from get_snapshot_storages to sort
* add tests
2021-06-02 18:24:55 -05:00
Jeff Washington (jwash)
72bb271a94
add metric for collecting storages ( #17527 )
2021-06-01 13:17:49 -05:00
Jeff Washington (jwash)
55c22d3b76
add check_hash to non-index hash calculation ( #17558 )
2021-05-28 10:24:40 -05:00
Brooks Prumo
1953543274
Refactor bank_forks_utils::load() and some snapshot logic ( #17492 )
...
Refactor a few functions that are on the load-from-snapshot path, to facilitate
adding in incremental snapshots more easily.
Additionally, add some tests and doc comments.
2021-05-27 12:00:27 -05:00
Jeff Washington (jwash)
6b9d8d41a3
add --limit_load_slot_count_from_snapshot to ledger-tool ( #17417 )
2021-05-26 10:36:12 -05:00
Jeff Washington (jwash)
0486df02ba
don't limit to thread pool when cleaning on startup ( #17317 )
2021-05-20 14:36:35 -05:00
Lijun Wang
9c42a89a43
Issue #17008 -- make snapshot archives to hold on to configurable. ( #17158 )
...
* purge_old_snapshot_archives is changed to take an extra argument 'maximum_snapshots_to_retain' to control the max number of latest snapshot archives to retain. Note the oldest snapshot is always retained as before and is not subjected to this new options.
* The validator and ledger-tool executables are modified with a CLI argument --maximum-snapshots-to-retain. And the options are propagated down the call chains. Their corresponding shell scripts were changed accordingly.
* SnapshotConfig is modified to have an extra field for the maximum_snapshots_to_retain
* Unit tests are developed to cover purge_old_snapshot_archives
2021-05-12 10:32:27 -07:00
Jeff Washington (jwash)
f39dda00e0
type AccountSecondaryIndexes = HashSet ( #17108 )
2021-05-10 14:22:48 +00:00
Stephen Akridge
3e0fed48e7
Don't recognize temp snapshots as possible snapshots to open
2021-05-05 08:45:03 -07:00
Brooks Prumo
17aa45fad1
Remove old way of account hashing
...
Account data hashing used to use different ways of hashing on different
clusters. That is no longer the case, but the old code still existed.
This commit removes that old, now used code.
**NOTE** The golden hash values in bank.rs needed to be updated. Since
the original code that selected the hash algorithm used `if >` instead
of `if >=`, this meant that the genesis block's hash _always_ used the
old hashing method, which is no longer valid.
Validated by running `cargo test` successfully.
2021-04-13 14:42:21 -05:00
Jeff Washington (jwash)
f374b35944
Hash stored accounts in bg ( #16157 )
...
* lazy calculate account hash
* push to bg thread
* remove deadlock
* logs
* format
* some cleanup on aisle 9
* format, fix up some metrics
* fix test, remove legacy function only there for tests
* cleanup
* remove unused store_hasher
* Switch to crossbeam
* clippy
* format
* use iter()
* rework from feedback
* hash_slot -> slot
* hash(cluster_type)
Co-authored-by: Carl Lin <carl@solana.com>
2021-03-31 15:39:34 -05:00
DimAn
a5d144b00f
Revert to removing only tmp-
2021-03-17 10:31:01 -07:00
DimAn
0209d334bd
Add option for separate snapshot location
...
(cherry picked from commit 6126878f509c69e23480a5ec22b3271e2b16e072)
2021-03-17 10:31:01 -07:00
Ryo Onodera
4bbeb9c033
Remove old feature: simple_capitalization ( #15763 )
...
* Remove old feature: simple_capitalization
* Fix another failing test in core
* Finish up test cleanup
* Further clean up a bit
2021-03-12 11:12:40 +09:00
Michael Vines
1061d021c9
Unpack snapshot AppendVecs directly into account paths
2021-03-11 08:25:24 -08:00
Michael Vines
65dd177601
AppendVec cleanup: remove dead code, rename non-idiomatic "new" function
2021-03-10 14:50:59 -08:00
Michael Vines
5df36aec7d
Pacify clippy
2021-02-19 20:08:41 -08:00
Jeff Washington (jwash)
fbf9dc47e9
Threadpool2 ( #15151 )
...
* rework thread pool for hash calculation
* rename
2021-02-05 18:48:16 -06:00
Jeff Washington (jwash)
fabecdc86c
use thread pool for non-index hash calculations ( #15149 )
2021-02-05 19:48:55 +00:00
Jeff Washington (jwash)
f0d58f5549
move timer end outside if ( #15087 )
2021-02-04 20:43:23 -06:00
Jeff Washington (jwash)
f49a70e626
borrow storages ( #15088 )
2021-02-04 11:29:03 -06:00
Jeff Washington (jwash)
600ff0d915
calculate hash from store instead of index ( #15034 )
...
* calculate hash from store instead of index
* restore update hash in abs
2021-02-04 09:00:33 -06:00
Michael Vines
cbffab7850
Upgrade to Rust v1.49.0
2021-01-23 19:16:36 -08:00
Michael Vines
bf1943e489
Add solana-test-validator --warp-slot argument
2021-01-22 21:17:02 -08:00
Michael Vines
dd5a2ef05f
Add convenience function to create a snapshot archive out of any Bank
2021-01-22 21:17:02 -08:00
carllin
6dfad0652f
Cache account stores, flush from AccountsBackgroundService ( #13140 )
2021-01-11 17:00:23 -08:00
Michael Vines
4a66e3eddc
Use standard tmp-snapshot- file prefix for the "new_state" archive for better cleanup/consistency
2021-01-11 16:51:26 -08:00
Michael Vines
872a249e97
Restore snapshot hard linking
2021-01-11 12:11:25 -08:00
Michael Vines
a95675a7ce
Avoid tmp snapshot backlog in SnapshotPackagerService under high load ( #14516 )
2021-01-11 10:21:15 -08:00
Michael Vines
7be6770808
Rename CompressionType to ArchiveFormat
2021-01-09 09:07:49 -08:00
Michael Vines
141e6706e6
Rename AccountsPackage::root to AccountsPackage::slot
2021-01-09 09:07:49 -08:00
Michael Vines
9f70f7dc3e
Reduce TempDir exposure
2021-01-09 09:07:49 -08:00
Michael Vines
f2a7f561a0
Replace incorrect symlink_dir usage with symlink_file
2021-01-09 09:07:49 -08:00
Michael Vines
db6ee289c9
Don't bother restoring snapshots, they're never used
2021-01-02 17:26:33 -08:00
carllin
5affd8aa72
Add secondary indexes ( #14212 )
2020-12-31 18:06:03 -08:00
Ryo Onodera
7893e2e307
Remove Testnet-specific old code ( #14305 )
2020-12-29 05:11:46 +09:00
Michael Vines
93ae177503
Do not delete ALL other snapshots before downloading a new snapshot
2020-12-21 08:58:52 +00:00
Michael Vines
73111b005f
Reduce the number of snapshots
2020-12-01 11:13:37 -08:00
Michael Vines
254790f8c8
Fully clean up temporary snapshot staging directories
2020-11-30 19:40:06 -08:00
Michael Vines
11e92f0c9f
Snapshot archives are now cleaned up when `--snapshot-compression none` is used
2020-11-20 08:41:17 -08:00
sakridge
43053dcc90
Re-use accounts stores ( #12885 )
...
* Re-use accounts_db stores
Creating files and dropping mmap areas can be expensive
* Add test for storage finder
Can encounter an infinite loop when the store is too small, but
smaller than the normal store size.
* Fix storage finding
* Check for strong_count == 1
* try_recycle helper
2020-11-04 09:17:05 -08:00
Ryo Onodera
e3773d919c
Avoid overflow when computing rent distribution ( #12112 )
...
* Avoid overflow when computing rent distribution
* Use assert_eq!....
* Fix tests
* Add test
* Use FeatureSet
* Add comments
* Address review comments
* Tweak a bit.
* Fix fmt
2020-10-01 11:59:28 +09:00
carllin
06f84c65f1
Fix rooted accounts cleanup, simplify locking ( #12194 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-28 16:04:46 -07:00
Michael Vines
31696a1d72
Port BPFLoader2 activation to FeatureSet and rework built-in program activation
2020-09-28 12:50:19 -07:00
Michael Vines
5dcf348098
Improve 'Failed to create snapshot archive' warning message
2020-09-25 20:02:49 +00:00
sakridge
68e5a2ef56
Add RPC notify and banking keys debug ( #12396 )
2020-09-23 18:46:42 -07:00
Ryo Onodera
5b2442d54e
Reduce cap by rent's leftover as temporary measure ( #12111 )
...
* Reduce cap by rent's leftover as temporary measure
* Reset testnet cap. on start and more logs
2020-09-09 03:53:48 +09:00
Michael Vines
d3750b47d2
Compress snapshot archive within the validator to reduce system dependencies
2020-09-06 22:48:12 -07:00
Michael Vines
d15173ad9d
Address latest nightly clippy lints, but globally disable stable_sort_primitive
2020-08-17 22:36:10 -07:00
Michael Vines
1d40685b8e
Remove support for 1.1.0 version snapshots
2020-08-04 01:01:50 +00:00
Kristofer Peterson
ed5a2f2a90
Fold bank serialisation into serde snapshot ( #10581 )
...
* Move bank (de)serialisation logic from bank and snapshot_utils to serde_snapshot.
Add sanity assertions between genesis config and bank fields on deserialisation.
Atomically update atomic bool in quote_for_specialization_detection().
Use same genesis config when restoring snapshots in test cases.
* Tidy up namings and duplicate structs to version
* Apply struct renames to tests
* Update abi hashes
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-07-13 23:00:59 +09:00
dependabot-preview[bot]
841ecfd927
chore(deps): bump bincode from 1.2.1 to 1.3.1 ( #10867 )
...
* chore(deps): bump bincode from 1.2.1 to 1.3.1
Bumps [bincode](https://github.com/servo/bincode ) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/servo/bincode/releases )
- [Commits](https://github.com/servo/bincode/commits )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* [auto-commit] Update all Cargo lock files
* Switch from deprecated method
* Add options to maintain behavior with bincode::options()
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-07-09 00:08:05 +00:00
Ryo Onodera
b89e506cbb
Consistently create temp dirs under ledger/farf ( #10848 )
2020-07-01 11:38:59 +09:00
sakridge
7430978d1a
Keep oldest snapshot so playing the ledger is possible from local data. ( #10814 )
2020-06-25 20:41:44 -07:00
Kristofer Peterson
6d81eede93
Add CLI options and runtime support for selection of output snapshot version. ( #10536 )
2020-06-19 14:38:37 +09:00
Greg Fitzgerald
6ee222363e
Move BankForks to solana_runtime ( #10637 )
...
* Move BankForks to solana_runtime
* Update imports
2020-06-17 15:27:03 +00:00