Jeff Washington (jwash)
1f00b468e5
add enable_rehashing to AccountsPackage ( #27644 )
2022-09-08 09:25:25 -07:00
Brooks Prumo
6a322de845
Make Accounts Background Services aware of Epoch Accounts Hash ( #27626 )
2022-09-07 20:41:40 +00:00
Jeff Washington (jwash)
a31d4a597d
serialize epoch_accounts_hash ( #27516 )
2022-09-07 10:07:00 -07:00
Brooks Prumo
a2df1e95dc
Hold `PendingAccountsPackage` lock for both checking and submitting ( #27613 )
...
Hold package lock for both checking and submitting
2022-09-06 20:11:13 +00:00
apfitzge
af971e63ab
Create all shared buffer readers before spawning threads ( #27512 )
...
* Create all shared buffer readers before spawning threads
* Allow 'needless' collect - clippy is wrong
* move clippy allow, spacing
2022-08-31 22:49:02 -05:00
apfitzge
49411aaae0
Feature: Interleaved Snapshot Untar and Indexing - Stage 1 ( #26590 )
...
* Add snapshot_storage_lengths_from_fields
* Add SnapshotUnpacker
* Add SnapshotStorageRebuilder
* Rebuild snapshot storage during unpack
* move snapshot unpacker back into snapshot_utils
* use SerializedAppendVecId in storage rebuilder
* create AccountStorageMap type alias
* Wrap storage and next_append_vec_id in a type
* fixed typo
* move use StorageAndNextAppendVecId
* calculate num_rebuilder_threads more concisely
2022-08-29 13:17:27 -05:00
Brennan Watt
e4a7d01e10
Rust v1.63 ( #27303 )
...
* Upgrade to Rust v1.63.0
* Add nightly_clippy_allows
* Resolve some new clippy nightly lints
* Increase QUIC packets completion timeout
* Update quinn-udp crate
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-22 18:01:03 -07:00
Brooks Prumo
510d195620
Simplify `Bank::clean_accounts()` by removing params ( #27254 )
2022-08-19 18:15:04 -04:00
Brennan Watt
7573000d87
Revert "Rust v1.63.0 ( #27148 )" ( #27245 )
...
This reverts commit a2e7bdf50a
.
2022-08-19 09:19:44 +01:00
Brooks Prumo
fa4122f9a8
Verifying snapshot bank must always specify the snapshot slot ( #27234 )
2022-08-18 17:24:23 +00:00
Brooks Prumo
d2868f439d
Verify snapshot slot deltas ( #26666 )
2022-08-18 09:48:58 -04:00
Brennan Watt
a2e7bdf50a
Rust v1.63.0 ( #27148 )
...
* Upgrade to Rust v1.63.0
* Add nightly_clippy_allows
* Resolve some new clippy nightly lints
* Increase QUIC packets completion timeout
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-17 15:48:33 -07:00
Jeff Washington (jwash)
225cddcffb
serialize incremental_snapshot_hash ( #26839 )
...
* serialize incremental_snapshot_hash
* pr feedback
2022-08-17 15:14:31 -05:00
apfitzge
c1111fa069
snapshots: serialize version file first ( #27192 )
...
serialize version file first
2022-08-17 12:57:52 -05:00
Justin Starry
69598ed4c0
Refactor: Add `RuntimeConfig` field to Bank ( #26946 )
...
* Refactor: Simplify arguments for bank constructor methods
* Refactor: Add RuntimeConfig to Bank fields
* Arc wrap runtime_config
* Arc wrap all runtime config usages
* Remove Copy trait derivation from RuntimeConfig
* Remove some arc wrapping
2022-08-05 20:49:00 +01:00
apfitzge
07f41628db
untar_snapshot_create_shared_buffer doesn't need to return a result ( #26686 )
2022-07-19 23:15:41 +00:00
apfitzge
e7cd6daebe
Add helper fn to create shared buffer - simplify archive format matching ( #26674 )
2022-07-19 20:30:30 +00:00
apfitzge
77843306c7
rename parallel_archivers -> parallel_divisions for consistency ( #26672 )
...
parallel_archivers -> parallel_divisions for consistency
2022-07-19 13:15:33 -05:00
Jeff Washington (jwash)
602da5e51f
add accounts db config to bank tests ( #26517 )
2022-07-10 19:42:06 -05:00
carllin
90ef2cd02a
Parse snapshot for bank fields ( #26016 )
2022-07-06 17:30:30 -05:00
Jeff Washington (jwash)
17a99d98dd
Revert "avoid adding to 'uncleaned_roots' when generating index and c… ( #26441 )
...
Revert "avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink (#25936 )"
This reverts commit e24cc537a4
.
2022-07-06 11:32:45 -05:00
Jeff Washington (jwash)
38216aa781
update comment for the next weary traveller ( #26413 )
2022-07-05 23:12:35 -05:00
Brooks Prumo
877fedadac
Remove StatusCacheRc type and use StatusCache directly ( #26184 )
2022-06-24 08:38:56 -05:00
Brooks Prumo
23c50a2389
Add StatusCache::root_slot_deltas() and use it ( #26170 )
2022-06-23 15:19:06 -05:00
Brooks Prumo
8a18e02f80
Call StatusCache::append() directly ( #26125 )
2022-06-22 07:42:01 -05:00
Tyera Eulberg
752c851070
Clean up feature preventing new rent-paying accounts ( #26000 )
...
* Clean up require_rent_exempt_accounts feature
* Fix tests in bank_client, system_instruction_processor
* Fix tests in snapshot_utils
* Fix bank tests
* Ignore rent tests
* Fix stake-accounts tests
2022-06-16 15:35:25 -06:00
Jeff Washington (jwash)
e24cc537a4
avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink ( #25936 )
2022-06-14 10:10:44 -05:00
apfitzge
3e1fd4dad0
snapshot_utils.rs: remove unnecessary sort ( #25893 )
2022-06-13 11:12:33 -05:00
apfitzge
04969a2282
serialize snapshots dir first ( #25889 )
2022-06-13 09:33:32 -05:00
apfitzge
56d38e4954
Add lz4 to snapshot filename regex for matching files on load ( #25633 )
...
* Add lz4 to snapshot filename regex for matching files on load
* Add tests for lz4 archive format filenames
2022-05-31 11:06:41 -05:00
HaoranYi
85a2e599bb
revert memory map for reading snapshot ( #25259 )
2022-05-17 09:32:18 -05:00
HaoranYi
3e44046a73
Support lz4 for snapshot archives ( #25089 )
...
* add lz4
* add cargo package file
* fix tests
* use fast lz4 compression
* report snapshot archive format
* add test
* code review feedback
* add cargo.lock
* fix var name
* refactor archive format parsing and add default compress cli arg
* clippy
* add from_cli_arg test
* update cargo.lock
* add lz4 support for mmap
* cargo.lock
* clippy
2022-05-16 12:44:15 -05:00
HaoranYi
389d78e424
Don't use mmarinus package ( #25177 )
...
* Revert "Revert "Use memory map to speed up snapshot untar (#24889 )" (#25174 )"
This reverts commit fc793de2dd
.
* not use mmarinus
* enable secondary build
* Revert "enable secondary build"
This reverts commit 5aa43a94bbad59497506f02235597e7adc0dc47b.
* macbuild
* Revert "macbuild"
This reverts commit 0da92945a40dc5431eff4bd91054b407e22418fb.
2022-05-16 08:49:17 -05:00
Jeff Washington (jwash)
fc793de2dd
Revert "Use memory map to speed up snapshot untar ( #24889 )" ( #25174 )
...
This reverts commit 3367e44671
.
2022-05-12 14:48:29 -05:00
HaoranYi
3367e44671
Use memory map to speed up snapshot untar ( #24889 )
...
* mmap
* add no_os_memory_reporting_args to ledger-tool
* use safe memory map package for unix. use regular file for windows
* fallback to untaring with filebuf when mmap fails
* update comments
2022-05-12 13:35:19 -05:00
DimAn
2fa9bc3e70
Add options to store full and/or incremental snapshots in separate locations ( #24247 )
2022-05-10 16:37:41 -04:00
HaoranYi
fd44198c0a
fix typo ( #24937 )
2022-05-03 09:57:15 -05:00
apfitzge
112a0b475a
Revert "Refactor to use EpochSchedule from within RentCollector struct" ( #24893 )
...
* Revert "Ran cargo fmt"
This reverts commit 9052e41b32
.
* Revert "Fix build error introduced by my editor setup, part 2"
This reverts commit 4dfeab3b38
.
* Revert "Fix build error introduced by my editor setup"
This reverts commit 87fb78dc56
.
* Revert "Remove redundant epoch_schedule from AccountsPackage"
This reverts commit c2f7f2fff8
.
* Revert "Fix a test"
This reverts commit 36c0bdaa78
.
* Revert "Fixes to initial code"
This reverts commit ed7813e698
.
* Revert "Removing redundant EpochSchedule param from fns"
This reverts commit 5472d2e605
.
2022-05-02 13:46:17 -05:00
Michael Vines
74b586ae71
Always retain the highest incremental snapshot for all full snapshots
2022-05-02 07:37:55 -07:00
Jeff Washington (jwash)
94d3ec37d2
collect_rent_eagerly can just do rewrites ( #24438 )
2022-04-18 22:44:19 -05:00
Jason Davis
c2f7f2fff8
Remove redundant epoch_schedule from AccountsPackage
2022-04-18 11:57:40 -05:00
Jeff Washington (jwash)
210f6a6fab
move hash calculation out of acct bg svc ( #23689 )
...
* move hash calculation out of acct bg svc
* pr feedback
2022-04-08 10:42:03 -05:00
Jeff Washington (jwash)
c27150b1a3
reserialize_bank_fields_with_hash ( #23916 )
...
* reserialize_bank_with_new_accounts_hash
* Update runtime/src/serde_snapshot.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-07 14:05:57 -05:00
Jeff Washington (jwash)
550ca7bf92
compare contents of serialized banks instead of exact file format ( #24141 )
...
* compare contents of serialized banks instead of exact file format
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
* get rid of clone
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-06 21:55:44 -05:00
Jeff Washington (jwash)
fddd162645
reserialize bank in ahv by first writing to temp file in abs ( #23947 )
2022-04-06 21:39:26 -05:00
Brooks Prumo
c322842257
Replace channel with Mutex<Option> for AccountsPackage ( #24013 )
2022-04-06 05:47:19 -05:00
Jeff Washington (jwash)
9c8dad33c7
add epoch_schedule and rent_collector to hash calc ( #24012 )
2022-03-31 10:51:18 -05:00
Brooks Prumo
31b707b625
Specify if archive size datapoint is for full or incremental snapshots ( #23941 )
2022-03-26 12:29:13 -05:00
Jeff Washington (jwash)
bc35e1c5f5
snapshot code needs all storages for hash calc ( #23840 )
2022-03-22 21:27:54 -05:00
HaoranYi
8c4f010b8d
fix logging: only walk remote dir if it exist ( #23663 )
2022-03-15 08:56:22 -05:00