Commit Graph

626 Commits

Author SHA1 Message Date
steviez f83f658cd8
ledger-tool: Remove duplicate open_genesis_config_by() call (#29950) 2023-01-27 01:20:42 -06:00
Brooks 685c22ff41
Inlines variables into format strings (#29945) 2023-01-27 06:23:03 +00:00
Xiang Zhu 4ebcacb4a3
Revert "Add run parent directory for accounts files (#29794)" (#29899)
This PR is causing OOM on master.  Reverting it for now.

This reverts commit 74f89d1494.
2023-01-25 10:03:01 -08:00
HaoranYi 2194551f87
Special case slot 0 for block connectness check in ledger tool (#29860)
* special case is_connected check for slot 0 so that we can load snapshot 0 for gce cluster boostrapting

* remove obsolete comments

* refactor with match expr

* update comments

* Update ledger-tool/src/main.rs

Co-authored-by: steviez <steven@solana.com>

* Update ledger-tool/src/main.rs

Co-authored-by: steviez <steven@solana.com>

Co-authored-by: steviez <steven@solana.com>
2023-01-25 09:06:22 -06:00
Yihau Chen 9193b4221d
Revert "chore: workspace inheritance (#29509)" (#29892)
This reverts commit a67d239dde.
2023-01-25 15:50:41 +08:00
Yihau Chen a67d239dde
chore: workspace inheritance (#29509)
* introduce workspace.package

* introduce workspace.dependencies

* read version from root cargo.toml

* pass check when version = { workspace = true }

* don't bump version when version = { workspace = true }

* including workspace Cargo.toml when bump version

* programs/sbf use workspace inheritance

* fix increasing cargo version ignore program/sbf/Cargo.toml
2023-01-25 13:59:59 +08:00
Xiang Zhu 74f89d1494
Add run parent directory for accounts files (#29794)
* Add run parent directory for accounts files

* fix test test_concurrent_snapshot_packaging

* review comments.  renamed the path setup function

* Addressed most of the review comments

* remove explict type def for map result

* handle create_accounts_run_and_snapshot_dirs error with expect

* update with more review comments

* minor fixes from review comments

* simplify account_filename option assignment

* handle error from create_accounts_run_and_snapshot_dirs

* use then instead of then_some for lazy evaluation
2023-01-24 16:44:35 -08:00
apfitzge 834b98aac0
Check that halt_slot >= starting_slot (#29868)
* Check that halt_slot >= starting_slot

* Improve message by saying where starting slot comes from
2023-01-24 14:38:52 -08:00
apfitzge 624f3d2d8c
Bugfix: bank-hash optionally takes halt_at_slot (#29865)
bank-hash optionally takes halt_at_slot
2023-01-24 14:15:07 -08:00
Trent Nelson c4e43f1de4
vote: encapsulate `Lockout` (#29753) 2023-01-18 19:28:28 -07:00
steviez 9bd0ce4082
ledger-tool: Clean account paths in all cases (#29609)
The existing logic would clean account paths only when we had secondary
access to the blockstore. However, the access mode shouldn't dictate
when we clean accounts data; we can and should clean account data from
previous runs in all instances given that we always start over from a
snapshot.
2023-01-17 21:53:51 -06:00
Giovanni Napoli 5eab3fb314
Add output in JSON for `solana-ledger-tool bounds` subcommand (#28410)
Introduce a struct to store all of the relevant slot/root information, and then output all in one go at the end as either human-readable or json. There are some slight changes to the human-readable format for the case of an empty ledger
2023-01-12 23:21:04 -06:00
Brooks d0aa93de21
Refactors accounts hash cache (#29625) 2023-01-12 10:43:50 -05:00
steviez 3992cee195
ledger-tool: Update dated println!() calls (#29636)
Removed an "ok" print that occurs after several commands; we already
print a log statement that indicates ledger-tool is done and how long it
took. Additionally, switch a println to info to avoid polluting stdout
incase we want to print information in a more easily readable format
such as json.
2023-01-10 20:41:47 -06:00
Jeff Washington (jwash) c9fe21e3c3
ancient append vecs enabled at -10k by default (#29565) 2023-01-10 05:33:43 -08:00
Yihau Chen 94cb88ffad
add solana-ledger-tool bigtable copy (#28122)
* init copy cmd

* extract creating emulator connection logic

* extract copy args as struct

* add new_for_emulator

* add tryFrom confirmed block to versioned block

* implement bigtable copy command

* use 'force' flag to force upload

* use unwrap_or

* remove redundant importing

* fix nightly lint

* explicit transactions missing error

* process ending_slot

* prevent start slot > end slot

* print skip slots in debug level

* fix destination bigtable should not be readonly

* combine is-emulator and endpoint to emulated source. conflict with crenditial path

* wording

* log some error messages with error level

* nightly lint

* add dry-run

* extract create bigtable instances logic

* use a lighter way to check block

* use the latest futures version which is used in the repo

* use futures = "0.3"

Co-authored-by: Tyera <teulberg@gmail.com>

* wording

Co-authored-by: Tyera <teulberg@gmail.com>

* wording

Co-authored-by: Tyera <teulberg@gmail.com>

Co-authored-by: Tyera <teulberg@gmail.com>
2023-01-09 11:23:35 +08:00
Boqin Qin(秦 伯钦)@MetaSecure cd5c5804ab
ledger-tool: fix double-readlock in main (#27322) 2023-01-06 19:30:07 +00:00
HaoranYi 4a8b980b81
update leger tool help for db verify refcounts (#29486) 2023-01-03 13:34:06 -06:00
Xiang Zhu 3363c08ac0
Move async remove to snapshot_utils.rs (#29406) 2023-01-03 06:15:32 -08:00
Jeff Washington (jwash) 4cc1890f00
skip_rewrites will only be feature driven (#29468) 2023-01-02 20:15:42 -08:00
steviez 1cc8917b9f
Adjust ledger-tool bigtable upload starting-slot default value (#29384)
Currently, if starting-slot is unspecified, a value of 0 will be chosen.
In the common case where someone is operating on a much more recent
range, this would result in a ton of wasted operations & time.

Instead, choose a smarter default value for starting-slot based on what
we detect is currently in the blockstore.
2022-12-23 21:35:07 +00:00
Tyera 81394cf92c
Fix BigTable upload early return (#29378)
* Change Err when slot range is empty to Ok and log; add method docs

* Update var and log to be more correct

* Promote log level to warn
2022-12-23 18:11:27 +00:00
Brooks Prumo 4530d161b7
Fixes format strings (#29327) 2022-12-19 17:26:38 -05:00
steviez 0609dce36c
Make ledger-tool list-roots subcommand inclusive of start-root arg (#29265) 2022-12-15 22:33:57 -06:00
Jeff Washington (jwash) b95835143e
remove AccountsBackgroundService::new(caching_enabled) (#29234) 2022-12-13 07:18:02 -08:00
Jeff Washington (jwash) fec8f61566
remove ProcessOptions::accounts_db_caching_enabled (#29217) 2022-12-12 20:25:00 -08:00
Brooks Prumo e02d1670fb
Uses Storages to calculate accounts hash when warping in `ledger-tool create-snapshot` (#29194) 2022-12-12 11:22:15 -05:00
Jeff Washington (jwash) 631a98a3b6
warp_from_parents works with write_cache enabled (#29185) 2022-12-09 14:28:18 -08:00
HaoranYi ca5d8c4b4d
refactor sysmonitor config (#29132) 2022-12-09 07:43:03 -06:00
Jeff Washington (jwash) 2f4731570a
ledger-tool create-snapshot uses write cache (#29119)
* ledger-tool create-snapshot uses write cache

* ledger tool create snapshot works with write cache
2022-12-08 06:09:52 -08:00
HaoranYi 33b15240ac
Revert #28945 (#29127)
revert #28945
2022-12-06 17:08:56 -06:00
behzad nouri 9524c9dbff patches errors from clippy::uninlined_format_args
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
haoran 037f03c806 fix 2022-12-06 13:31:50 -06:00
haoran 1f7d2785ef fix build 2022-12-06 13:31:50 -06:00
Haoran Yi e1ba5a2a63 add monitoring for open file descriptors stat 2022-12-06 13:31:50 -06:00
Maximilian Schneider c8b0c3ede9
Update cost model to use requested_cu instead of estimated cu #27608 (#28281)
* Update cost model to use requested_cu instead of estimated cu #27608

* remove CostUpdate and CostModel from replay/tvu

* revive cost update service to send cost tracker stats

* CostModel is now static

* remove unused package

Co-authored-by: Tao Zhu <tao@solana.com>
2022-11-22 11:55:56 -06:00
apfitzge 08e017b5ba
Improve error message (#28898) 2022-11-22 10:02:44 -06:00
Jeff Washington (jwash) f0ef316b19
remove cli arg "--no-accounts-db-caching" (#28912) 2022-11-21 12:38:35 -08:00
steviez 7bc78196b5
ledger-tool: collapse repeated encoding argument parsing (#28859) 2022-11-21 13:15:14 -05:00
Trent Nelson 53a579bed1 ledger-tool: stream output of `accounts` subcommand 2022-11-17 12:45:26 -07:00
Jeff Washington (jwash) f69e847137
allow ancient append vec threshold to be set on cli (#28707)
* allow ancient append vec threshold to be set on cli

* pr responses
2022-11-11 11:30:05 -08:00
Brooks Prumo d1ba42180d
clippy for rust 1.65.0 (#28765) 2022-11-09 19:39:38 +00:00
steviez 54fc768fcc
Single source ledger-tool encoding arg (#28622)
The argument is nearly identical for both instances that use it, so
de-duplicate the repeated config.
2022-10-27 07:35:27 +00:00
Trent Nelson 2c0831aa12 ledger-tool: remove inefficient base58 encoding options 2022-10-19 19:16:52 -07:00
Brooks Prumo dd7fee8f32
Re-enqueues unhandled ABS requests (#28362) 2022-10-13 16:25:39 -04:00
Brooks Prumo 9cbd00fdbc
Converts PendingAccountsPackage to a channel (#28352) 2022-10-13 12:47:36 -04:00
Jeff Washington (jwash) 7ae7a3ff0e
after replay at validator startup don't verify hash calc (#28222) 2022-10-04 14:04:04 -07:00
steviez 7fef7d569a
Make RocksFifo storage size API expose an Option<u64> (#28192)
A fifo rocksdb instance must be opened with max size parameter on the
fifo columns. To support this, we previously plumbed a constant up to
callers that provided a default if unbounded growth desired.

This change attempts to be more rusty by exposing an option for this
value, and converting the option to a constant at the lowest level
possible.
2022-10-04 12:38:40 -05:00
steviez 8cc8a0656a
Cleanup solana-ledger-tool slot output (#28129) 2022-10-03 23:25:44 -05:00
Yueh-Hsuan Chiang 6b17bee5a8
Remove the const default for RocksFifo (#27965)
#### Summary of Changes
Removes the constant default for ShredStorageType::RocksFifo
as the shred storage size is either user-specified or derived
from --limit-ledger-size in #27459.
2022-10-01 15:10:54 -07:00