Commit Graph

390 Commits

Author SHA1 Message Date
Brooks d0aa93de21
Refactors accounts hash cache (#29625) 2023-01-12 10:43:50 -05:00
steviez 3992cee195
ledger-tool: Update dated println!() calls (#29636)
Removed an "ok" print that occurs after several commands; we already
print a log statement that indicates ledger-tool is done and how long it
took. Additionally, switch a println to info to avoid polluting stdout
incase we want to print information in a more easily readable format
such as json.
2023-01-10 20:41:47 -06:00
Jeff Washington (jwash) c9fe21e3c3
ancient append vecs enabled at -10k by default (#29565) 2023-01-10 05:33:43 -08:00
Boqin Qin(秦 伯钦)@MetaSecure cd5c5804ab
ledger-tool: fix double-readlock in main (#27322) 2023-01-06 19:30:07 +00:00
HaoranYi 4a8b980b81
update leger tool help for db verify refcounts (#29486) 2023-01-03 13:34:06 -06:00
Xiang Zhu 3363c08ac0
Move async remove to snapshot_utils.rs (#29406) 2023-01-03 06:15:32 -08:00
Jeff Washington (jwash) 4cc1890f00
skip_rewrites will only be feature driven (#29468) 2023-01-02 20:15:42 -08:00
Brooks Prumo 4530d161b7
Fixes format strings (#29327) 2022-12-19 17:26:38 -05:00
steviez 0609dce36c
Make ledger-tool list-roots subcommand inclusive of start-root arg (#29265) 2022-12-15 22:33:57 -06:00
Jeff Washington (jwash) b95835143e
remove AccountsBackgroundService::new(caching_enabled) (#29234) 2022-12-13 07:18:02 -08:00
Jeff Washington (jwash) fec8f61566
remove ProcessOptions::accounts_db_caching_enabled (#29217) 2022-12-12 20:25:00 -08:00
Brooks Prumo e02d1670fb
Uses Storages to calculate accounts hash when warping in `ledger-tool create-snapshot` (#29194) 2022-12-12 11:22:15 -05:00
Jeff Washington (jwash) 631a98a3b6
warp_from_parents works with write_cache enabled (#29185) 2022-12-09 14:28:18 -08:00
HaoranYi ca5d8c4b4d
refactor sysmonitor config (#29132) 2022-12-09 07:43:03 -06:00
Jeff Washington (jwash) 2f4731570a
ledger-tool create-snapshot uses write cache (#29119)
* ledger-tool create-snapshot uses write cache

* ledger tool create snapshot works with write cache
2022-12-08 06:09:52 -08:00
HaoranYi 33b15240ac
Revert #28945 (#29127)
revert #28945
2022-12-06 17:08:56 -06:00
behzad nouri 9524c9dbff patches errors from clippy::uninlined_format_args
https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
2022-12-06 19:32:15 +00:00
haoran 037f03c806 fix 2022-12-06 13:31:50 -06:00
haoran 1f7d2785ef fix build 2022-12-06 13:31:50 -06:00
Haoran Yi e1ba5a2a63 add monitoring for open file descriptors stat 2022-12-06 13:31:50 -06:00
Maximilian Schneider c8b0c3ede9
Update cost model to use requested_cu instead of estimated cu #27608 (#28281)
* Update cost model to use requested_cu instead of estimated cu #27608

* remove CostUpdate and CostModel from replay/tvu

* revive cost update service to send cost tracker stats

* CostModel is now static

* remove unused package

Co-authored-by: Tao Zhu <tao@solana.com>
2022-11-22 11:55:56 -06:00
apfitzge 08e017b5ba
Improve error message (#28898) 2022-11-22 10:02:44 -06:00
Jeff Washington (jwash) f0ef316b19
remove cli arg "--no-accounts-db-caching" (#28912) 2022-11-21 12:38:35 -08:00
steviez 7bc78196b5
ledger-tool: collapse repeated encoding argument parsing (#28859) 2022-11-21 13:15:14 -05:00
Trent Nelson 53a579bed1 ledger-tool: stream output of `accounts` subcommand 2022-11-17 12:45:26 -07:00
Jeff Washington (jwash) f69e847137
allow ancient append vec threshold to be set on cli (#28707)
* allow ancient append vec threshold to be set on cli

* pr responses
2022-11-11 11:30:05 -08:00
Brooks Prumo d1ba42180d
clippy for rust 1.65.0 (#28765) 2022-11-09 19:39:38 +00:00
steviez 54fc768fcc
Single source ledger-tool encoding arg (#28622)
The argument is nearly identical for both instances that use it, so
de-duplicate the repeated config.
2022-10-27 07:35:27 +00:00
Trent Nelson 2c0831aa12 ledger-tool: remove inefficient base58 encoding options 2022-10-19 19:16:52 -07:00
Brooks Prumo dd7fee8f32
Re-enqueues unhandled ABS requests (#28362) 2022-10-13 16:25:39 -04:00
Brooks Prumo 9cbd00fdbc
Converts PendingAccountsPackage to a channel (#28352) 2022-10-13 12:47:36 -04:00
Jeff Washington (jwash) 7ae7a3ff0e
after replay at validator startup don't verify hash calc (#28222) 2022-10-04 14:04:04 -07:00
steviez 7fef7d569a
Make RocksFifo storage size API expose an Option<u64> (#28192)
A fifo rocksdb instance must be opened with max size parameter on the
fifo columns. To support this, we previously plumbed a constant up to
callers that provided a default if unbounded growth desired.

This change attempts to be more rusty by exposing an option for this
value, and converting the option to a constant at the lowest level
possible.
2022-10-04 12:38:40 -05:00
steviez 8cc8a0656a
Cleanup solana-ledger-tool slot output (#28129) 2022-10-03 23:25:44 -05:00
Yueh-Hsuan Chiang 6b17bee5a8
Remove the const default for RocksFifo (#27965)
#### Summary of Changes
Removes the constant default for ShredStorageType::RocksFifo
as the shred storage size is either user-specified or derived
from --limit-ledger-size in #27459.
2022-10-01 15:10:54 -07:00
Yueh-Hsuan Chiang 599677f965
Derive rocksdb_fifo_shred_storage_size based on limit-ledger-size (#27459)
### Problem
When FIFO compaction is used while --rocksdb_fifo_shred_storage_size
is unspecified, the FIFO shred storage size is set to a const default based
on the default `--limit-ledger-size`.

### Summary of the Change
When --rocksdb_fifo_shred_storage_size is unspecified, it is now
derived from `--limit-ledger-size` by reserving 1500 bytes for each
shred.
2022-09-28 00:32:27 -07:00
Amit Panghal 860cd607fd
add flag to genesis cmd to dump account data in specified format (#27695)
* feat: add flag to genesis cmd to dump account data in jsonParsed format

* update Cargo.lock

* add json-parsed flag

* fix formatting

* fix formatting again

* use rustfmt

* use --encoding

* fix for failing checks

* Apply suggestions from code review

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* address review comments

* address review comments

* fix indent

* fix print fmt

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2022-09-18 22:52:53 -06:00
steviez 1d5314b5e6
Make solana-ledger-tool retain all snapshots by default (#27788) 2022-09-15 16:21:12 -05:00
Yueh-Hsuan Chiang 70c4f6ea96
Enable ledger-tool copy command to infer target_db shred storage type. (#27438)
#### Problem
The ledger-tool copy command currently assumes target_db uses the same
shred storage type as the source ledger.

#### Summary of Changes
This PR enables ledger-tool copy command to infer the target_db shred storage
type based on whether rocksdb or rocksdb_fifo exists under the target_db directory
(same way as how ledger-tool infers the shred storage type for the main db.)
If the ledger-tool is not able to infer the shred storage type of the target_db, then
the default level compaction will be used.
2022-09-13 10:27:58 -07:00
Jeff Washington (jwash) 765c628546
use exit signal for acct idx bg threads (#27483) 2022-09-12 11:51:12 -07:00
Jeff Washington (jwash) 518e16744d
move --accounts-db-verify-refcounts under verify (#27651) 2022-09-08 07:59:24 -07:00
Xiang Zhu 4308c300b4
In ledger-tool delete the account files in the async way (#27622)
* In ledger-tool delete the account files in the async way

* format changes by ./cargo nightly fmt --all
2022-09-07 14:35:06 -07:00
Brooks Prumo 93a4f80a2c
Handling snapshot requests is now required (#27537) 2022-09-07 10:08:42 -04:00
Yueh-Hsuan Chiang ed00365101
Add ledger tool command print-file-metadata (#26790)
Add ledger-tool command print-file-metadata

#### Summary of Changes
This PR adds a ledger tool subcommand print-file-metadata.
```
USAGE:
    solana-ledger-tool print-file-metadata [FLAGS] [OPTIONS] [SST_FILE_NAME]

    Prints the metadata of the specified ledger-store file.
    If no file name is unspecified, then it will print the metadata of all ledger files
```
2022-09-06 21:46:35 -07:00
Jeff Washington (jwash) 72a2c02ee7
add --accounts-db-verify-refcounts for debugging (#27504) 2022-09-06 09:39:39 -07:00
Brooks Prumo 6684c62280
Add SnapshotUsage to SnapshotConfig (#27508) 2022-09-02 08:56:23 -04:00
Brooks Prumo 3c7cd62030
Move pruned_banks_receiver into PrunedBanksRequestHandler (#27445) 2022-08-29 13:30:06 -04:00
apfitzge 6da3eb028c
create-snapshot check if snapshot slot exists (#27153) 2022-08-19 09:34:35 -05:00
apfitzge 40b9f2f2be
slots_connected: check if the range is connected (>= ending_slot) (#27152) 2022-08-19 09:33:50 -05:00
steviez adcc36da8e
Add fallback for ledger-tool commands to create new column families (#26565)
RocksDB settings include an option to create_if_missing, which will
create missing columns or the entire rocksdb directory if starting from
scratch. However, create_if_missing functionality only works if the
session has Primary (read+write) access. Many ledger-tool commands only
need Secondary (read-only) access to the database, so these commands are
unable to open the Blockstore when a column must be added.

This change detects when Secondary access open fails due to missing
column(s) or files, opens the database temporarily with Primary access,
and then reattempts to open the database Secondary access.
2022-08-14 05:20:13 -05:00