Commit Graph

633 Commits

Author SHA1 Message Date
steviez 37461976e0
ledger-tool: Move shred storage type inference next to Blockstore::open (#30084) 2023-02-02 16:55:18 -06:00
steviez 90f8c86d23
ledger-tool: Remove unnecessary function to parse ledger path arg (#29952) 2023-02-01 13:01:58 -06:00
Will Hickey 04a6a631bc
Bump version to v1.16 (#30028) 2023-01-31 17:48:33 -06:00
joeaba a12bf8c003
Update maintainers references (#29997)
* update maintainers references

* chore: update maintainers reference
2023-01-31 08:07:13 -05:00
steviez 1e6e623da5
ledger-tool: Move AccountsDbConfig building to common helper function (#29975)
Several subcommands may wish to specify AccountsDb configuration
options. The matching is currently duplicated in both "verify" and
"create-snapshot" command, so move the config to a common helper.

Note that each subcommand that wishes to use any argument will still
need to add the argument to its' argument list under App::new().
2023-01-30 20:59:35 +00:00
Xiang Zhu 856598969c
Account path add run parent with old path cleanup (#29942)
* Add run parent directory for accounts files

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* remove explict type def for map result

* handle create_accounts_run_and_snapshot_dirs error with expect

* update with more review comments

* minor fixes from review comments

* simplify account_filename option assignment

* handle error from create_accounts_run_and_snapshot_dirs

* use then instead of then_some for lazy evaluation

* Clean up files in the old account_path before trasitioning to the new run path

* try_exist and accounts_dir removing extra

* sync rmdir, is_dir check

* handle the account_path not deletable case
2023-01-30 10:26:43 -08:00
steviez 76ae62596a
ledger-tool: Fix argument matching for accounts_db_ancient_append_vecs (#29976)
The logic was searching for argument in global argument matches when it
should have been searching in the subcommand matches.
2023-01-30 10:56:07 -06:00
steviez f83f658cd8
ledger-tool: Remove duplicate open_genesis_config_by() call (#29950) 2023-01-27 01:20:42 -06:00
Brooks 685c22ff41
Inlines variables into format strings (#29945) 2023-01-27 06:23:03 +00:00
Xiang Zhu 4ebcacb4a3
Revert "Add run parent directory for accounts files (#29794)" (#29899)
This PR is causing OOM on master.  Reverting it for now.

This reverts commit 74f89d1494.
2023-01-25 10:03:01 -08:00
HaoranYi 2194551f87
Special case slot 0 for block connectness check in ledger tool (#29860)
* special case is_connected check for slot 0 so that we can load snapshot 0 for gce cluster boostrapting

* remove obsolete comments

* refactor with match expr

* update comments

* Update ledger-tool/src/main.rs

Co-authored-by: steviez <steven@solana.com>

* Update ledger-tool/src/main.rs

Co-authored-by: steviez <steven@solana.com>

Co-authored-by: steviez <steven@solana.com>
2023-01-25 09:06:22 -06:00
Yihau Chen 9193b4221d
Revert "chore: workspace inheritance (#29509)" (#29892)
This reverts commit a67d239dde.
2023-01-25 15:50:41 +08:00
Yihau Chen a67d239dde
chore: workspace inheritance (#29509)
* introduce workspace.package

* introduce workspace.dependencies

* read version from root cargo.toml

* pass check when version = { workspace = true }

* don't bump version when version = { workspace = true }

* including workspace Cargo.toml when bump version

* programs/sbf use workspace inheritance

* fix increasing cargo version ignore program/sbf/Cargo.toml
2023-01-25 13:59:59 +08:00
Xiang Zhu 74f89d1494
Add run parent directory for accounts files (#29794)
* Add run parent directory for accounts files

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* remove explict type def for map result

* handle create_accounts_run_and_snapshot_dirs error with expect

* update with more review comments

* minor fixes from review comments

* simplify account_filename option assignment

* handle error from create_accounts_run_and_snapshot_dirs

* use then instead of then_some for lazy evaluation
2023-01-24 16:44:35 -08:00
apfitzge 834b98aac0
Check that halt_slot >= starting_slot (#29868)
* Check that halt_slot >= starting_slot

* Improve message by saying where starting slot comes from
2023-01-24 14:38:52 -08:00
apfitzge 624f3d2d8c
Bugfix: bank-hash optionally takes halt_at_slot (#29865)
bank-hash optionally takes halt_at_slot
2023-01-24 14:15:07 -08:00
Trent Nelson c4e43f1de4
vote: encapsulate `Lockout` (#29753) 2023-01-18 19:28:28 -07:00
steviez 9bd0ce4082
ledger-tool: Clean account paths in all cases (#29609)
The existing logic would clean account paths only when we had secondary
access to the blockstore. However, the access mode shouldn't dictate
when we clean accounts data; we can and should clean account data from
previous runs in all instances given that we always start over from a
snapshot.
2023-01-17 21:53:51 -06:00
Giovanni Napoli 5eab3fb314
Add output in JSON for `solana-ledger-tool bounds` subcommand (#28410)
Introduce a struct to store all of the relevant slot/root information, and then output all in one go at the end as either human-readable or json. There are some slight changes to the human-readable format for the case of an empty ledger
2023-01-12 23:21:04 -06:00
Brooks d0aa93de21
Refactors accounts hash cache (#29625) 2023-01-12 10:43:50 -05:00
steviez 3992cee195
ledger-tool: Update dated println!() calls (#29636)
Removed an "ok" print that occurs after several commands; we already
print a log statement that indicates ledger-tool is done and how long it
took. Additionally, switch a println to info to avoid polluting stdout
incase we want to print information in a more easily readable format
such as json.
2023-01-10 20:41:47 -06:00
Jeff Washington (jwash) c9fe21e3c3
ancient append vecs enabled at -10k by default (#29565) 2023-01-10 05:33:43 -08:00
Yihau Chen 94cb88ffad
add solana-ledger-tool bigtable copy (#28122)
* init copy cmd

* extract creating emulator connection logic

* extract copy args as struct

* add new_for_emulator

* add tryFrom confirmed block to versioned block

* implement bigtable copy command

* use 'force' flag to force upload

* use unwrap_or

* remove redundant importing

* fix nightly lint

* explicit transactions missing error

* process ending_slot

* prevent start slot > end slot

* print skip slots in debug level

* fix destination bigtable should not be readonly

* combine is-emulator and endpoint to emulated source. conflict with crenditial path

* wording

* log some error messages with error level

* nightly lint

* add dry-run

* extract create bigtable instances logic

* use a lighter way to check block

* use the latest futures version which is used in the repo

* use futures = "0.3"

Co-authored-by: Tyera <teulberg@gmail.com>

* wording

Co-authored-by: Tyera <teulberg@gmail.com>

* wording

Co-authored-by: Tyera <teulberg@gmail.com>

Co-authored-by: Tyera <teulberg@gmail.com>
2023-01-09 11:23:35 +08:00
Boqin Qin(秦 伯钦)@MetaSecure cd5c5804ab
ledger-tool: fix double-readlock in main (#27322) 2023-01-06 19:30:07 +00:00
HaoranYi 4a8b980b81
update leger tool help for db verify refcounts (#29486) 2023-01-03 13:34:06 -06:00
Xiang Zhu 3363c08ac0
Move async remove to snapshot_utils.rs (#29406) 2023-01-03 06:15:32 -08:00
Jeff Washington (jwash) 4cc1890f00
skip_rewrites will only be feature driven (#29468) 2023-01-02 20:15:42 -08:00
steviez 1cc8917b9f
Adjust ledger-tool bigtable upload starting-slot default value (#29384)
Currently, if starting-slot is unspecified, a value of 0 will be chosen.
In the common case where someone is operating on a much more recent
range, this would result in a ton of wasted operations & time.

Instead, choose a smarter default value for starting-slot based on what
we detect is currently in the blockstore.
2022-12-23 21:35:07 +00:00
Tyera 81394cf92c
Fix BigTable upload early return (#29378)
* Change Err when slot range is empty to Ok and log; add method docs

* Update var and log to be more correct

* Promote log level to warn
2022-12-23 18:11:27 +00:00
Brooks Prumo 4530d161b7
Fixes format strings (#29327) 2022-12-19 17:26:38 -05:00
steviez 0609dce36c
Make ledger-tool list-roots subcommand inclusive of start-root arg (#29265) 2022-12-15 22:33:57 -06:00
Jeff Washington (jwash) b95835143e
remove AccountsBackgroundService::new(caching_enabled) (#29234) 2022-12-13 07:18:02 -08:00
Jeff Washington (jwash) fec8f61566
remove ProcessOptions::accounts_db_caching_enabled (#29217) 2022-12-12 20:25:00 -08:00
Brooks Prumo e02d1670fb
Uses Storages to calculate accounts hash when warping in `ledger-tool create-snapshot` (#29194) 2022-12-12 11:22:15 -05:00
Jeff Washington (jwash) 631a98a3b6
warp_from_parents works with write_cache enabled (#29185) 2022-12-09 14:28:18 -08:00
HaoranYi ca5d8c4b4d
refactor sysmonitor config (#29132) 2022-12-09 07:43:03 -06:00
Jeff Washington (jwash) 2f4731570a
ledger-tool create-snapshot uses write cache (#29119)
* ledger-tool create-snapshot uses write cache

* ledger tool create snapshot works with write cache
2022-12-08 06:09:52 -08:00
HaoranYi 33b15240ac
Revert #28945 (#29127)
revert #28945
2022-12-06 17:08:56 -06:00
behzad nouri 9524c9dbff patches errors from clippy::uninlined_format_args
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
haoran 037f03c806 fix 2022-12-06 13:31:50 -06:00
haoran 1f7d2785ef fix build 2022-12-06 13:31:50 -06:00
Haoran Yi e1ba5a2a63 add monitoring for open file descriptors stat 2022-12-06 13:31:50 -06:00
Maximilian Schneider c8b0c3ede9
Update cost model to use requested_cu instead of estimated cu #27608 (#28281)
* Update cost model to use requested_cu instead of estimated cu #27608

* remove CostUpdate and CostModel from replay/tvu

* revive cost update service to send cost tracker stats

* CostModel is now static

* remove unused package

Co-authored-by: Tao Zhu <tao@solana.com>
2022-11-22 11:55:56 -06:00
apfitzge 08e017b5ba
Improve error message (#28898) 2022-11-22 10:02:44 -06:00
Jeff Washington (jwash) f0ef316b19
remove cli arg "--no-accounts-db-caching" (#28912) 2022-11-21 12:38:35 -08:00
steviez 7bc78196b5
ledger-tool: collapse repeated encoding argument parsing (#28859) 2022-11-21 13:15:14 -05:00
Trent Nelson 53a579bed1 ledger-tool: stream output of `accounts` subcommand 2022-11-17 12:45:26 -07:00
Jeff Washington (jwash) f69e847137
allow ancient append vec threshold to be set on cli (#28707)
* allow ancient append vec threshold to be set on cli

* pr responses
2022-11-11 11:30:05 -08:00
Brooks Prumo d1ba42180d
clippy for rust 1.65.0 (#28765) 2022-11-09 19:39:38 +00:00
steviez 54fc768fcc
Single source ledger-tool encoding arg (#28622)
The argument is nearly identical for both instances that use it, so
de-duplicate the repeated config.
2022-10-27 07:35:27 +00:00