Brooks Prumo
8a18e02f80
Call StatusCache::append() directly ( #26125 )
2022-06-22 07:42:01 -05:00
Tyera Eulberg
752c851070
Clean up feature preventing new rent-paying accounts ( #26000 )
...
* Clean up require_rent_exempt_accounts feature
* Fix tests in bank_client, system_instruction_processor
* Fix tests in snapshot_utils
* Fix bank tests
* Ignore rent tests
* Fix stake-accounts tests
2022-06-16 15:35:25 -06:00
Jeff Washington (jwash)
e24cc537a4
avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink ( #25936 )
2022-06-14 10:10:44 -05:00
apfitzge
3e1fd4dad0
snapshot_utils.rs: remove unnecessary sort ( #25893 )
2022-06-13 11:12:33 -05:00
apfitzge
04969a2282
serialize snapshots dir first ( #25889 )
2022-06-13 09:33:32 -05:00
apfitzge
56d38e4954
Add lz4 to snapshot filename regex for matching files on load ( #25633 )
...
* Add lz4 to snapshot filename regex for matching files on load
* Add tests for lz4 archive format filenames
2022-05-31 11:06:41 -05:00
HaoranYi
85a2e599bb
revert memory map for reading snapshot ( #25259 )
2022-05-17 09:32:18 -05:00
HaoranYi
3e44046a73
Support lz4 for snapshot archives ( #25089 )
...
* add lz4
* add cargo package file
* fix tests
* use fast lz4 compression
* report snapshot archive format
* add test
* code review feedback
* add cargo.lock
* fix var name
* refactor archive format parsing and add default compress cli arg
* clippy
* add from_cli_arg test
* update cargo.lock
* add lz4 support for mmap
* cargo.lock
* clippy
2022-05-16 12:44:15 -05:00
HaoranYi
389d78e424
Don't use mmarinus package ( #25177 )
...
* Revert "Revert "Use memory map to speed up snapshot untar (#24889 )" (#25174 )"
This reverts commit fc793de2dd
.
* not use mmarinus
* enable secondary build
* Revert "enable secondary build"
This reverts commit 5aa43a94bbad59497506f02235597e7adc0dc47b.
* macbuild
* Revert "macbuild"
This reverts commit 0da92945a40dc5431eff4bd91054b407e22418fb.
2022-05-16 08:49:17 -05:00
Jeff Washington (jwash)
fc793de2dd
Revert "Use memory map to speed up snapshot untar ( #24889 )" ( #25174 )
...
This reverts commit 3367e44671
.
2022-05-12 14:48:29 -05:00
HaoranYi
3367e44671
Use memory map to speed up snapshot untar ( #24889 )
...
* mmap
* add no_os_memory_reporting_args to ledger-tool
* use safe memory map package for unix. use regular file for windows
* fallback to untaring with filebuf when mmap fails
* update comments
2022-05-12 13:35:19 -05:00
DimAn
2fa9bc3e70
Add options to store full and/or incremental snapshots in separate locations ( #24247 )
2022-05-10 16:37:41 -04:00
HaoranYi
fd44198c0a
fix typo ( #24937 )
2022-05-03 09:57:15 -05:00
apfitzge
112a0b475a
Revert "Refactor to use EpochSchedule from within RentCollector struct" ( #24893 )
...
* Revert "Ran cargo fmt"
This reverts commit 9052e41b32
.
* Revert "Fix build error introduced by my editor setup, part 2"
This reverts commit 4dfeab3b38
.
* Revert "Fix build error introduced by my editor setup"
This reverts commit 87fb78dc56
.
* Revert "Remove redundant epoch_schedule from AccountsPackage"
This reverts commit c2f7f2fff8
.
* Revert "Fix a test"
This reverts commit 36c0bdaa78
.
* Revert "Fixes to initial code"
This reverts commit ed7813e698
.
* Revert "Removing redundant EpochSchedule param from fns"
This reverts commit 5472d2e605
.
2022-05-02 13:46:17 -05:00
Michael Vines
74b586ae71
Always retain the highest incremental snapshot for all full snapshots
2022-05-02 07:37:55 -07:00
Jeff Washington (jwash)
94d3ec37d2
collect_rent_eagerly can just do rewrites ( #24438 )
2022-04-18 22:44:19 -05:00
Jason Davis
c2f7f2fff8
Remove redundant epoch_schedule from AccountsPackage
2022-04-18 11:57:40 -05:00
Jeff Washington (jwash)
210f6a6fab
move hash calculation out of acct bg svc ( #23689 )
...
* move hash calculation out of acct bg svc
* pr feedback
2022-04-08 10:42:03 -05:00
Jeff Washington (jwash)
c27150b1a3
reserialize_bank_fields_with_hash ( #23916 )
...
* reserialize_bank_with_new_accounts_hash
* Update runtime/src/serde_snapshot.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-07 14:05:57 -05:00
Jeff Washington (jwash)
550ca7bf92
compare contents of serialized banks instead of exact file format ( #24141 )
...
* compare contents of serialized banks instead of exact file format
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
* get rid of clone
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-06 21:55:44 -05:00
Jeff Washington (jwash)
fddd162645
reserialize bank in ahv by first writing to temp file in abs ( #23947 )
2022-04-06 21:39:26 -05:00
Brooks Prumo
c322842257
Replace channel with Mutex<Option> for AccountsPackage ( #24013 )
2022-04-06 05:47:19 -05:00
Jeff Washington (jwash)
9c8dad33c7
add epoch_schedule and rent_collector to hash calc ( #24012 )
2022-03-31 10:51:18 -05:00
Brooks Prumo
31b707b625
Specify if archive size datapoint is for full or incremental snapshots ( #23941 )
2022-03-26 12:29:13 -05:00
Jeff Washington (jwash)
bc35e1c5f5
snapshot code needs all storages for hash calc ( #23840 )
2022-03-22 21:27:54 -05:00
HaoranYi
8c4f010b8d
fix logging: only walk remote dir if it exist ( #23663 )
2022-03-15 08:56:22 -05:00
HaoranYi
0c684721d8
Separate remotely downloaded snapshot archives ( #23510 )
...
* seperate remotely downloaded snapshot archives
* add str const for snapshot download dir
* only walk remote sub directory
* move directory creation outside of loop
* move is_remote to traits
* clippy simplify
* clippy
* clippy
* add unittest
* fix local cluster tests
* look for remote snapshot archive in remote foler
* create remote dir in tests
* use snapshot download dir constant
* extract build_remote_dir fn
* fix build
* code review - walking snapshot archived dirs explicitly
* fix build
* fix build
* fix comments
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* clippy
* borrow to avoid copy
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-03-14 14:03:59 -05:00
Giorgio Gambino
ccff006948
Add more logs to `purge_old_snapshot_archives` ( #23531 )
2022-03-11 15:01:48 -06:00
Brooks Prumo
1eddb6d1e9
Move ArchiveFormat into own module ( #23562 )
2022-03-09 16:09:34 -06:00
Brooks Prumo
9bbccbe27c
Use `AsRef<Path>` instead of `PathBuf` for parameters ( #23560 )
2022-03-09 16:08:33 -06:00
Michael Vines
5599bd9442
Remove BankFromArchiveTimings from ledger/
2022-03-08 08:11:50 -08:00
HaoranYi
181fffb916
rename status filename to be consistent ( #23501 )
2022-03-07 17:34:35 +00:00
HaoranYi
463cd564cf
fix typos ( #23495 )
...
* fix typos
2022-03-05 20:46:46 -06:00
Michael Vines
b28acd2d4d
Reduce fn visibility
2022-03-04 09:52:46 +01:00
Michael Vines
72c68695b5
Do not keep oldest full snapshot
2022-02-21 17:10:31 -07:00
Michael Vines
577fa4ec0c
Generate full snapshots 4x faster to keep incremental snapshots nice and small
2022-02-15 21:09:21 -08:00
Brooks Prumo
b82d71d22a
Inline DEFAULT_SNAPSHOT_VERSION constant string ( #22487 )
2022-01-13 18:19:15 +00:00
Brooks Prumo
2756abce39
More serde snapshot cleanup ( #22449 )
2022-01-13 09:20:20 -06:00
mooori
0f6e8d3385
Check file size of `snapshot_version` when unarchiving snapshot ( #21925 )
2021-12-17 12:27:54 -06:00
Michael Vines
b8837c04ec
Reformat imports to a consistent style for imports
...
rustfmt.toml configuration:
imports_granularity = "One"
group_imports = "One"
2021-12-03 09:19:13 -08:00
Michael Vines
ba9dfa0d22
Remove frozen account support
2021-11-29 08:38:11 -08:00
Jack May
bced07a099
Update fee api to use blockhash ( #21054 )
2021-10-29 13:52:59 -07:00
Justin Starry
036d7fcc81
Clean up sanitized tx creation for tests ( #21006 )
2021-10-27 18:09:16 +01:00
Jack May
bfbbc53dac
Divorce the runtime from FeeCalculator ( #20737 )
2021-10-22 14:32:40 -07:00
Brooks Prumo
751b45df64
Make pub get_incremental_snapshot_archives() ( #20725 )
2021-10-18 13:48:50 -05:00
Jack May
da45be366a
Remove blockhash from fee calculation ( #20641 )
2021-10-13 13:10:58 -07:00
Michael Vines
7027d56064
Resolve nightly-2021-10-05 clippy complaints
2021-10-06 10:37:58 -07:00
Lijun Wang
fe97cb2ddf
AccountsDb plugin framework ( #20047 )
...
Summary of Changes
Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows
Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.
The code comprises 4 major parts:
accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-09-30 14:26:17 -07:00
Brooks Prumo
95fbc29500
Add stats when getting snapshot storages ( #20234 )
2021-09-27 13:07:00 -05:00
Brooks Prumo
21bc7e94f5
Add logging of incremental snapshots to purge_old_snapshot_archives() ( #19874 )
2021-09-14 17:49:15 -05:00