07f41628db
untar_snapshot_create_shared_buffer doesn't need to return a result ( #26686 )
2022-07-19 23:15:41 +00:00
e7cd6daebe
Add helper fn to create shared buffer - simplify archive format matching ( #26674 )
2022-07-19 20:30:30 +00:00
77843306c7
rename parallel_archivers -> parallel_divisions for consistency ( #26672 )
...
parallel_archivers -> parallel_divisions for consistency
2022-07-19 13:15:33 -05:00
602da5e51f
add accounts db config to bank tests ( #26517 )
2022-07-10 19:42:06 -05:00
90ef2cd02a
Parse snapshot for bank fields ( #26016 )
2022-07-06 17:30:30 -05:00
17a99d98dd
Revert "avoid adding to 'uncleaned_roots' when generating index and c… ( #26441 )
...
Revert "avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink (#25936 )"
This reverts commit e24cc537a4
2022-07-06 11:32:45 -05:00
38216aa781
update comment for the next weary traveller ( #26413 )
2022-07-05 23:12:35 -05:00
877fedadac
Remove StatusCacheRc type and use StatusCache directly ( #26184 )
2022-06-24 08:38:56 -05:00
23c50a2389
Add StatusCache::root_slot_deltas() and use it ( #26170 )
2022-06-23 15:19:06 -05:00
8a18e02f80
Call StatusCache::append() directly ( #26125 )
2022-06-22 07:42:01 -05:00
752c851070
Clean up feature preventing new rent-paying accounts ( #26000 )
...
* Clean up require_rent_exempt_accounts feature
* Fix tests in bank_client, system_instruction_processor
* Fix tests in snapshot_utils
* Fix bank tests
* Ignore rent tests
* Fix stake-accounts tests
2022-06-16 15:35:25 -06:00
e24cc537a4
avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink ( #25936 )
2022-06-14 10:10:44 -05:00
3e1fd4dad0
snapshot_utils.rs: remove unnecessary sort ( #25893 )
2022-06-13 11:12:33 -05:00
04969a2282
serialize snapshots dir first ( #25889 )
2022-06-13 09:33:32 -05:00
56d38e4954
Add lz4 to snapshot filename regex for matching files on load ( #25633 )
...
* Add lz4 to snapshot filename regex for matching files on load
* Add tests for lz4 archive format filenames
2022-05-31 11:06:41 -05:00
85a2e599bb
revert memory map for reading snapshot ( #25259 )
2022-05-17 09:32:18 -05:00
3e44046a73
Support lz4 for snapshot archives ( #25089 )
...
* add lz4
* add cargo package file
* fix tests
* use fast lz4 compression
* report snapshot archive format
* add test
* code review feedback
* add cargo.lock
* fix var name
* refactor archive format parsing and add default compress cli arg
* clippy
* add from_cli_arg test
* update cargo.lock
* add lz4 support for mmap
* cargo.lock
* clippy
2022-05-16 12:44:15 -05:00
389d78e424
Don't use mmarinus package ( #25177 )
...
* Revert "Revert "Use memory map to speed up snapshot untar (#24889 )" (#25174 )"
This reverts commit fc793de2dd
2022-05-16 08:49:17 -05:00
fc793de2dd
Revert "Use memory map to speed up snapshot untar ( #24889 )" ( #25174 )
...
This reverts commit 3367e44671
2022-05-12 14:48:29 -05:00
3367e44671
Use memory map to speed up snapshot untar ( #24889 )
...
* mmap
* add no_os_memory_reporting_args to ledger-tool
* use safe memory map package for unix. use regular file for windows
* fallback to untaring with filebuf when mmap fails
* update comments
2022-05-12 13:35:19 -05:00
2fa9bc3e70
Add options to store full and/or incremental snapshots in separate locations ( #24247 )
2022-05-10 16:37:41 -04:00
fd44198c0a
fix typo ( #24937 )
2022-05-03 09:57:15 -05:00
112a0b475a
Revert "Refactor to use EpochSchedule from within RentCollector struct" ( #24893 )
...
* Revert "Ran cargo fmt"
This reverts commit 9052e41b324dfeab3b3887fb78dc56c2f7f2fff836c0bdaa78ed7813e6985472d2e605
2022-05-02 13:46:17 -05:00
74b586ae71
Always retain the highest incremental snapshot for all full snapshots
2022-05-02 07:37:55 -07:00
94d3ec37d2
collect_rent_eagerly can just do rewrites ( #24438 )
2022-04-18 22:44:19 -05:00
c2f7f2fff8
Remove redundant epoch_schedule from AccountsPackage
2022-04-18 11:57:40 -05:00
210f6a6fab
move hash calculation out of acct bg svc ( #23689 )
...
* move hash calculation out of acct bg svc
* pr feedback
2022-04-08 10:42:03 -05:00
c27150b1a3
reserialize_bank_fields_with_hash ( #23916 )
...
* reserialize_bank_with_new_accounts_hash
* Update runtime/src/serde_snapshot.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-07 14:05:57 -05:00
550ca7bf92
compare contents of serialized banks instead of exact file format ( #24141 )
...
* compare contents of serialized banks instead of exact file format
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
* get rid of clone
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-06 21:55:44 -05:00
fddd162645
reserialize bank in ahv by first writing to temp file in abs ( #23947 )
2022-04-06 21:39:26 -05:00
c322842257
Replace channel with Mutex<Option> for AccountsPackage ( #24013 )
2022-04-06 05:47:19 -05:00
9c8dad33c7
add epoch_schedule and rent_collector to hash calc ( #24012 )
2022-03-31 10:51:18 -05:00
31b707b625
Specify if archive size datapoint is for full or incremental snapshots ( #23941 )
2022-03-26 12:29:13 -05:00
bc35e1c5f5
snapshot code needs all storages for hash calc ( #23840 )
2022-03-22 21:27:54 -05:00
8c4f010b8d
fix logging: only walk remote dir if it exist ( #23663 )
2022-03-15 08:56:22 -05:00
0c684721d8
Separate remotely downloaded snapshot archives ( #23510 )
...
* seperate remotely downloaded snapshot archives
* add str const for snapshot download dir
* only walk remote sub directory
* move directory creation outside of loop
* move is_remote to traits
* clippy simplify
* clippy
* clippy
* add unittest
* fix local cluster tests
* look for remote snapshot archive in remote foler
* create remote dir in tests
* use snapshot download dir constant
* extract build_remote_dir fn
* fix build
* code review - walking snapshot archived dirs explicitly
* fix build
* fix build
* fix comments
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* clippy
* borrow to avoid copy
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-03-14 14:03:59 -05:00
ccff006948
Add more logs to `purge_old_snapshot_archives` ( #23531 )
2022-03-11 15:01:48 -06:00
1eddb6d1e9
Move ArchiveFormat into own module ( #23562 )
2022-03-09 16:09:34 -06:00
9bbccbe27c
Use `AsRef<Path>` instead of `PathBuf` for parameters ( #23560 )
2022-03-09 16:08:33 -06:00
5599bd9442
Remove BankFromArchiveTimings from ledger/
2022-03-08 08:11:50 -08:00
181fffb916
rename status filename to be consistent ( #23501 )
2022-03-07 17:34:35 +00:00
463cd564cf
fix typos ( #23495 )
...
* fix typos
2022-03-05 20:46:46 -06:00
b28acd2d4d
Reduce fn visibility
2022-03-04 09:52:46 +01:00
72c68695b5
Do not keep oldest full snapshot
2022-02-21 17:10:31 -07:00
577fa4ec0c
Generate full snapshots 4x faster to keep incremental snapshots nice and small
2022-02-15 21:09:21 -08:00
b82d71d22a
Inline DEFAULT_SNAPSHOT_VERSION constant string ( #22487 )
2022-01-13 18:19:15 +00:00
2756abce39
More serde snapshot cleanup ( #22449 )
2022-01-13 09:20:20 -06:00
0f6e8d3385
Check file size of `snapshot_version` when unarchiving snapshot ( #21925 )
2021-12-17 12:27:54 -06:00
b8837c04ec
Reformat imports to a consistent style for imports
...
rustfmt.toml configuration:
imports_granularity = "One"
group_imports = "One"
2021-12-03 09:19:13 -08:00
ba9dfa0d22
Remove frozen account support
2021-11-29 08:38:11 -08:00
apfitzge
Jeff Washington (jwash)
carllin
Brooks Prumo
Tyera Eulberg
HaoranYi
Jeff Washington (jwash)
DimAn
Michael Vines
Jason Davis
Giorgio Gambino
mooori