Commit Graph

13 Commits

Author SHA1 Message Date
Xiang Zhu d69f60229d
Construct bank from snapshot dir (#30171)
* Construct a bank from a snapshot directory

* Clean up.  Remove archiving releated data structures

* Fix auto checks

* remove ArchiveFormat::None

* fix merge error

* remove incremental snapshot dir

* Minor cleanup, remove unused functiond defs

* remove pub fn bank_from_latest_snapshot_dir

* rename bank_from_snapshot_dir to bank_from_snapshot

* Clean up invalid comments

* A few minor review changes

* Removed insert_slot_storage_file

* Add comment explain hardlink symlink

* Skip the whole verify_snapshot_bank call for the from_dir case

* Add bank.set_initial_accounts_hash_verification_completed()

* address review issues: appendvec to append_vec, replace unwrap with expect, etc

* AtomicAppendVecId, remove arc on bank etc

* slice, CI error on &snapshot_version_path

* measure_build_storage

* move snapshot_from

* remove measure_name from build_storage_from_snapshot_dir

* remove from_dir specific next_append_vec_id logic

* revert insert_slot_storage_file change

* init next_append_vec_id to fix the substraction underflow

* remove measure from build_storage_from_snapshot_dir

* make measure name more specific

* refactor status_cache deserialization into a function

* remove reference to pass the ci check

* track next appendvec id

* verify that the next_append_vec_id tracking is correct

* clean up usize

* in build_storage_from_snapshot_dir remove expect

* test max appendvecc id tracking with multiple banks in the test

* cleared expect and unwrap in streaming_snapshot_dir_files

* rebase cleanup

* change to measure!

* dereference arc in the right way
2023-03-22 13:49:23 -07:00
Jeff Washington (jwash) b26d470b02
support loading snapshots with > 1 append vec per slot (#30570)
* support loading snapshots with > 1 append vec per slot

* pr feedback

* drain to into_values

* unwrap() -> expect()

* loop to find non-conflicting file and add test
2023-03-06 06:08:57 -08:00
Jeff Washington (jwash) 461dafb887
AccountStorage only holds a single append vec (#29723)
* AccountStorageReference

* tests build

* pr feedback
2023-01-16 16:22:30 -08:00
Xiang Zhu a0f045d2c3
storage rebuilder regex cleanup (#29408)
* storage rebuilder regex cleanup

* Update runtime/src/snapshot_utils/snapshot_storage_rebuilder.rs

Co-authored-by: apfitzge <apfitzge@gmail.com>

Co-authored-by: apfitzge <apfitzge@gmail.com>
2023-01-05 16:16:24 -08:00
Jeff Washington (jwash) 49a136a374
move AccountStorage out of accounts_db.rs (#29261) 2022-12-14 15:08:39 -08:00
apfitzge 151727f633
Storage Rebuilder: better error propagation (#29150) 2022-12-09 10:00:12 -06:00
behzad nouri 9524c9dbff patches errors from clippy::uninlined_format_args
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
Brooks Prumo d1ba42180d
clippy for rust 1.65.0 (#28765) 2022-11-09 19:39:38 +00:00
apfitzge 64f64dedea
Use regex in get_snapshot_file_kind (#27728)
refactor get_snapshot_file_kind to use regex
2022-09-22 13:55:48 -05:00
apfitzge a846d50b6a
Interleaved snapshot unpack versioning (#27484)
* Issue #27346 - deserialize and check snapshot version before account fields

* Update comment on SnapshotFileKind

Co-authored-by: Brooks Prumo <brooks@prumo.org>

* SnapshotStorageRebuilderResult to RebuiltSnapshotStorage

* better error propagation from rebuild_storage

Co-authored-by: apfitzge <apfitzge@users.noreply.github.com>
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-09-22 11:44:25 -05:00
apfitzge 49411aaae0
Feature: Interleaved Snapshot Untar and Indexing - Stage 1 (#26590)
* Add snapshot_storage_lengths_from_fields

* Add SnapshotUnpacker

* Add SnapshotStorageRebuilder

* Rebuild snapshot storage during unpack

* move snapshot unpacker back into snapshot_utils

* use SerializedAppendVecId in storage rebuilder

* create AccountStorageMap type alias

* Wrap storage and next_append_vec_id in a type

* fixed typo

* move use StorageAndNextAppendVecId

* calculate num_rebuilder_threads more concisely
2022-08-29 13:17:27 -05:00
HaoranYi 3e44046a73
Support lz4 for snapshot archives (#25089)
* add lz4

* add cargo package file

* fix tests

* use fast lz4 compression

* report snapshot archive format

* add test

* code review feedback

* add cargo.lock

* fix var name

* refactor archive format parsing and add default compress cli arg

* clippy

* add from_cli_arg test

* update cargo.lock

* add lz4 support for mmap

* cargo.lock

* clippy
2022-05-16 12:44:15 -05:00
Brooks Prumo 1eddb6d1e9
Move ArchiveFormat into own module (#23562) 2022-03-09 16:09:34 -06:00