70c4f6ea96
Enable ledger-tool copy command to infer target_db shred storage type. ( #27438 )
...
#### Problem
The ledger-tool copy command currently assumes target_db uses the same
shred storage type as the source ledger.
#### Summary of Changes
This PR enables ledger-tool copy command to infer the target_db shred storage
type based on whether rocksdb or rocksdb_fifo exists under the target_db directory
(same way as how ledger-tool infers the shred storage type for the main db.)
If the ledger-tool is not able to infer the shred storage type of the target_db, then
the default level compaction will be used.
2022-09-13 10:27:58 -07:00
c0e4379f43
Whickey/version v1.15 ( #27739 )
...
* Bump version to v1.13.0
* Bump version to v1.14.0
* Bump version to v1.15.0
2022-09-13 09:06:15 -05:00
765c628546
use exit signal for acct idx bg threads ( #27483 )
2022-09-12 11:51:12 -07:00
54284955de
Respect rewards parameter in getBlock transaction-level rewards ( #27655 )
...
* Respect options.show_rewards in UiTransactionStatusMeta
* Fixup docs
2022-09-08 10:17:02 -06:00
518e16744d
move --accounts-db-verify-refcounts under verify ( #27651 )
2022-09-08 07:59:24 -07:00
4308c300b4
In ledger-tool delete the account files in the async way ( #27622 )
...
* In ledger-tool delete the account files in the async way
* format changes by ./cargo nightly fmt --all
2022-09-07 14:35:06 -07:00
93a4f80a2c
Handling snapshot requests is now required ( #27537 )
2022-09-07 10:08:42 -04:00
ed00365101
Add ledger tool command print-file-metadata ( #26790 )
...
Add ledger-tool command print-file-metadata
#### Summary of Changes
This PR adds a ledger tool subcommand print-file-metadata.
```
USAGE:
solana-ledger-tool print-file-metadata [FLAGS] [OPTIONS] [SST_FILE_NAME]
Prints the metadata of the specified ledger-store file.
If no file name is unspecified, then it will print the metadata of all ledger files
```
2022-09-06 21:46:35 -07:00
72a2c02ee7
add --accounts-db-verify-refcounts for debugging ( #27504 )
2022-09-06 09:39:39 -07:00
6684c62280
Add SnapshotUsage to SnapshotConfig ( #27508 )
2022-09-02 08:56:23 -04:00
66717ff87d
chore: bump chrono from 0.4.21 to 0.4.22 ( #27509 )
...
* chore: bump chrono from 0.4.21 to 0.4.22
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.21 to 0.4.22.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/v0.4.22/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.21...v0.4.22 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-08-31 22:39:12 +00:00
3c7cd62030
Move pruned_banks_receiver into PrunedBanksRequestHandler ( #27445 )
2022-08-29 13:30:06 -04:00
6da3eb028c
create-snapshot check if snapshot slot exists ( #27153 )
2022-08-19 09:34:35 -05:00
40b9f2f2be
slots_connected: check if the range is connected (>= ending_slot) ( #27152 )
2022-08-19 09:33:50 -05:00
adcc36da8e
Add fallback for ledger-tool commands to create new column families ( #26565 )
...
RocksDB settings include an option to create_if_missing, which will
create missing columns or the entire rocksdb directory if starting from
scratch. However, create_if_missing functionality only works if the
session has Primary (read+write) access. Many ledger-tool commands only
need Secondary (read-only) access to the database, so these commands are
unable to open the Blockstore when a column must be added.
This change detects when Secondary access open fails due to missing
column(s) or files, opens the database temporarily with Primary access,
and then reattempts to open the database Secondary access.
2022-08-14 05:20:13 -05:00
f641d3bad6
chore: bump chrono from 0.4.19 to 0.4.21 ( #27076 )
...
* chore: bump chrono from 0.4.19 to 0.4.21
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.19 to 0.4.21.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.19...v0.4.21 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2022-08-11 09:28:50 -06:00
e3a8d2ecdd
chore: bump serde_json from 1.0.81 to 1.0.83 ( #27036 )
...
* chore: bump serde_json from 1.0.81 to 1.0.83
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.81 to 1.0.83.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.81...v1.0.83 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-08-10 09:45:42 -06:00
f7c6901191
ledger-tool: support Geyser accounts updates ( #26909 )
2022-08-09 15:44:57 +00:00
07a0076fc2
chore: bump bytecount from 0.6.2 to 0.6.3 ( #27014 )
...
Bumps [bytecount](https://github.com/llogiq/bytecount ) from 0.6.2 to 0.6.3.
- [Release notes](https://github.com/llogiq/bytecount/releases )
- [Commits](https://github.com/llogiq/bytecount/commits )
---
updated-dependencies:
- dependency-name: bytecount
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-09 00:33:43 -06:00
ed8c224374
Bump version to v1.12 ( #26967 )
2022-08-06 13:20:30 -05:00
66919e5adb
Unpin tokio for non-rpc crates ( #26957 )
2022-08-05 20:50:45 -06:00
300666dce7
Make `solana-ledger-tool` run AccountsBackgroundService ( #26914 )
...
Prior to this change, long running commands like `solana-ledger-tool
verify` would OOM due to AccountsDb cleanup not happening.
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-04 15:44:31 -05:00
fbf1bf6d86
Bump Version to 1.11.6 ( #26906 )
...
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
2022-08-03 12:48:43 -05:00
f3b760dd91
Add IO metrics ( #26804 )
...
* Add Disk IO metrics
2022-08-02 14:29:53 -07:00
c7462b7a52
ledger tool verify can store debug info on hash calc ( #26837 )
2022-07-29 15:54:56 -05:00
5d038b9d2a
Bump Version to 1.11.5 ( #26758 )
...
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
2022-07-25 13:05:14 -06:00
0fff5b8bf8
ledger-tool: optionally `graph` vote states
2022-07-22 17:43:01 -06:00
633950e029
ledger-tool: factor out `graph` options to config struct
2022-07-22 17:43:01 -06:00
a541827fb9
ledger-tool: fix off-by-one in `graph` skipped slot label
2022-07-22 17:43:01 -06:00
dcbda2c6c5
Allow ledger tool to automatically detect the shred compaction style ( #26182 )
...
#### Problem
Ledger-tool doesn't support shred-compaction-type other than the default rocksdb level compaction.
#### Summary of Changes
This PR enables ledger-tool to automatically detect the shred-compaction-type of the specified ledger.
#### Test Plan
New ledger-tool tests are added for both level and fifo compactions.
2022-07-19 01:19:17 +08:00
fd5df1cf25
Bump Version to 1.11.4 ( #26578 )
...
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
2022-07-11 23:30:38 -05:00
1c2f6a4c41
Bugfix: slots_connected snapshot_slot not full ( #26506 )
...
* slots_connected check used by ledger-tool should not require a full slot for snapshot slot
* Cleaner Result<Option<>> unwrap/default
* return false if no meta for starting slot
* Add clarifying comments
2022-07-11 13:45:12 -05:00
9d937fb8a0
Bump Version to 1.11.3 ( #26481 )
...
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
2022-07-07 14:39:46 -05:00
26176af4aa
cleanup cli arg help ( #26366 )
2022-07-05 14:49:05 -05:00
95ae82e074
add info to ledger tool for total execution time ( #26368 )
2022-07-05 09:23:06 -05:00
2b21c1da29
Sort Cargo.toml files
2022-06-24 12:41:38 -07:00
5c2f819f99
Bump Version to 1.11.2 ( #26159 )
2022-06-22 21:16:18 -05:00
f4189c0305
ledger-tool minimized snapshots ( #25334 )
...
* working on local snapshot
* Parallelization for slot storage minimization
* Additional clean-up and fixes
* make --minimize an option of create-snapshot
* remove now unnecessary function
* Parallelize parts of minimized account set generation
* clippy fixes
* Add rent collection accounts and voting node_pubkeys
* Simplify programdata_accounts generation
* Loop over storages to get slot set
* Parallelize minimized slot set generation
* Parallelize adding owners and programdata_accounts
* Remove some now unncessary checks on the blockstore
* Add a warning for minimized snapshots across epoch boundary
* Simplify ledger-tool minimize
* Clarify names of bank's minimization helper functions
* Remove unnecesary funciton, fix line spacing
* Use DashSets instead of HashSets for minimized account and slot sets
* Filter storages uses all threads instead of thread_pool
* Add some additional comments on functions for minimization
* Moved more into bank and parallelized
* Update programs/bpf/Cargo.lock for dashmap in ledger
* Clippy fix
* ledger-tool: convert minimize_bank_for_snapshot Measure into measure!
* bank.rs: convert minimize_bank_for_snapshot Measure into measure!
* accounts_db.rs: convert minimize_accounts_db Measure into measure!
* accounts_db.rs: add comment about use of minimize_accounts_db
* ledger-tool: CLI argument clarification
* minimization functions: make infos unique
* bank.rs: Add test_get_rent_collection_accounts_between_slots
* bank.rs: Add test_minimization_add_vote_accounts
* bank.rs: Add test_minimization_add_stake_accounts
* bank.rs: Add test_minimization_add_owner_accounts
* bank.rs: Add test_minimization_add_programdata_accounts
* accounts_db.rs: Add test_minimize_accounts_db
* bank.rs: Add negative case and comments in test_get_rent_collection_accounts_between_slots
* bank.rs: Negative test in test_minimization_add_programdata_accounts
* use new static runtime and sdk ids
* bank comments to doc comments
* Only need to insert the maximum slot a key is found in
* rename remove_pubkeys to purge_pubkeys
* add comment on builtins::get_pubkeys
* prevent excessive logging of removed dead slots
* don't need to remove slot from shrink slot candidates
* blockstore.rs: get_accounts_used_in_range shouldn't return Result
* blockstore.rs: get_accounts_used_in_range: parallelize slot loop
* report filtering progress on time instead of count
* parallelize loop over snapshot storages
* WIP: move some bank minimization functionality into a new class
* WIP: move some accounts_db minimization functionality into SnapshotMinimizer
* WIP: Use new SnapshotMinimizer
* SnapshotMinimizer: fix use statements
* remove bank and accounts_db minimization code, where possible
* measure! doesn't take a closure
* fix use statement in blockstore
* log_dead_slots does not need pub(crate)
* get_unique_accounts_from_storages does not need pub(crate)
* different way to get stake accounts/nodes
* fix tests
* move rent collection account functionality to snapshot minimizer
* move accounts_db minimize behavior to snapshot minimizer
* clean up
* Use bank reference instead of Arc. Additional comments
* Add a comment to blockstore function
* Additional clarifying comments
* Moved all non-transaction account accumulation into the SnapshotMinimizer.
* transaction_account_set does not need to be mutable now
* Add comment about load_to_collect_rent_eagerly
* Update log_dead_slots comment
* remove duplicate measure/print of get_minimized_slot_set
2022-06-22 13:17:43 -04:00
6513bb3b1b
add options to create snapshot ( #26095 )
2022-06-21 13:07:05 -05:00
51f26dc96e
Bump version to 1.11.1 ( #26104 )
2022-06-21 12:07:46 -05:00
2866ca4b1c
Add ledger-tool bigtable upload loop ( #26030 )
...
* Add ledger-tool bigtable upload loop
* Limit range on caller side, switch to while loop, and remove now-obsolete option
2022-06-17 19:31:13 +00:00
a9c9acfe80
Improve repair-roots help ( #26008 )
2022-06-16 10:03:27 -06:00
32a58dd9e0
Add ability to use a non-default app profile id in bigtable requests ( #25968 )
...
* Add ability to use a non-default app profile id in bigtable requests
* Only run subcommand once when getting global configs
* Remove unneded scoping on option type
2022-06-16 00:58:16 +00:00
ba04063956
Add CPUmetrics ( #25802 )
...
Add in some CPU utilization metrics such as: number of vCPUs, clock frequency, average load across different time intervals, and number of total threads
2022-06-07 11:34:25 -07:00
934da5ef99
Fix pre-check of blockstore slots during load_bank_forks ( #25632 )
...
Fix pre-check of blockstore slts during load_bank_forks. Now iterates from starting_slot to halt_slot via slot_meta.next_slots to confirm they are connected.
2022-06-01 20:19:42 -05:00
5b67960c76
(Refactor) Move blocktore options related stuff to blockstore_options.rs ( #25509 )
...
#### Problem
blockstore_db.rs has a mutual dependency between blockstore_metrics.rs.
#### Summary of Changes
This PR removes the mutual dependency by moving the option-related stuff
out from blockstore_db.rs to its new home --- blockstore_options.rs.
By doing this, we address the mutual dependency and also make the code cleaner.
2022-05-26 16:59:26 -07:00
328cca9fcd
Add a --debug-key option to ledger-tool verify ( #25391 )
2022-05-24 19:56:35 -05:00
61c5a471e8
preserve optimistic_slot in blockstore ( #25311 )
2022-05-24 12:03:28 -07:00
2cb500a388
fix problem with matches in ledger-tool ( #25482 )
2022-05-23 12:39:58 -05:00
9d9773bd2a
Use write! instead of format! to pacify clippy
2022-05-22 22:22:21 -07:00
b05c7d91ed
Fix derive_partial_eq_without_eq clippy lint
2022-05-22 22:22:21 -07:00
d65aab4d27
Fix ledger-tool slot to output tx even when tx status is missing ( #25434 )
2022-05-20 20:56:06 -06:00
467431de89
add ledger-tool accounts-db-skip-initial-hash-calc ( #25354 )
2022-05-20 10:27:00 -05:00
1e7d877494
Create-snapshot check if slot is available on startup ( #25329 )
...
* Add a check during ledger-tool create-snapshot startup to see if the snapshot slot is available
* check all slots from the start to snapshot_slot during load_bank_forks
* unwrap_or_default incremental snapshot slot before comparison
* Improve error messages on missing or not full slots
2022-05-20 09:11:18 -05:00
cdac141417
Sort per-program and print a summary of all slots in the range ( #25267 )
2022-05-19 23:26:29 +02:00
83b245642f
--snapshot-archive-format should not conflict w/ --no-snapshot ( #25368 )
2022-05-19 12:49:59 -05:00
3b47250d4a
chore: bump signal-hook from 0.3.13 to 0.3.14 ( #25281 )
...
* chore: bump signal-hook from 0.3.13 to 0.3.14
Bumps [signal-hook](https://github.com/vorner/signal-hook ) from 0.3.13 to 0.3.14.
- [Release notes](https://github.com/vorner/signal-hook/releases )
- [Changelog](https://github.com/vorner/signal-hook/blob/master/CHANGELOG.md )
- [Commits](https://github.com/vorner/signal-hook/compare/v0.3.13...v0.3.14 )
---
updated-dependencies:
- dependency-name: signal-hook
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-05-18 09:28:52 -06:00
3d8e5e91ed
Fix ledger-tool from erroring with unspecified archive_format ( #25298 )
...
value_t_or_exit()! will error if the user doesn't specify a value at
runtime, use value_of() instead which will give either the default value
or whatever the user specified.
2022-05-17 15:46:31 -06:00
3e44046a73
Support lz4 for snapshot archives ( #25089 )
...
* add lz4
* add cargo package file
* fix tests
* use fast lz4 compression
* report snapshot archive format
* add test
* code review feedback
* add cargo.lock
* fix var name
* refactor archive format parsing and add default compress cli arg
* clippy
* add from_cli_arg test
* update cargo.lock
* add lz4 support for mmap
* cargo.lock
* clippy
2022-05-16 12:44:15 -05:00
bc005e3408
Add configurable limit to number of blocks to check before Bigtable upload ( #24716 )
...
* Add ConfirmedBlockUploadConfig, no behavior changes
* Add comment
* A little DRY cleanup
* Add configurable limit to number of blocks to check in Blockstore and Bigtable before uploading
* Limit blockstore and bigtable look-ahead
* Exit iterator early when reach ending_slot
* Use rooted_slot_iterator instead of slot_meta_iterator
* Only check blocks in the ledger
2022-05-13 07:34:02 +00:00
a8930ee14b
add --accounts-db-ancient-append-vecs ( #25125 )
2022-05-11 08:47:07 -05:00
2fa9bc3e70
Add options to store full and/or incremental snapshots in separate locations ( #24247 )
2022-05-10 16:37:41 -04:00
b83fd8b958
ledger-tool: clear non-primary access accounts directory in load_bank_forks ( #24958 )
2022-05-09 13:17:13 -05:00
082502d4f3
Fail tx sanitization when ix program id uses lookup table ( #25035 )
...
* Fail tx sanitization when ix program id uses lookup table
* feedback
2022-05-07 03:19:50 +08:00
c4aa082efc
Rollback tokio to LTS release v1.14 ( #25028 )
2022-05-06 06:34:49 +00:00
bece7f32c8
chore: bump log from 0.4.16 to 0.4.17 ( #24987 )
...
* chore: bump log from 0.4.16 to 0.4.17
Bumps [log](https://github.com/rust-lang/log ) from 0.4.16 to 0.4.17.
- [Release notes](https://github.com/rust-lang/log/releases )
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/log/commits/0.4.17 )
---
updated-dependencies:
- dependency-name: log
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-05-05 01:49:18 +00:00
61a9faae17
chore: bump serde_json from 1.0.80 to 1.0.81 ( #24960 )
...
* chore: bump serde_json from 1.0.80 to 1.0.81
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.80 to 1.0.81.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.80...v1.0.81 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2022-05-04 14:57:38 -06:00
2c9d2a2140
chore: bump serde_json from 1.0.79 to 1.0.80 ( #24943 )
...
* chore: bump serde_json from 1.0.79 to 1.0.80
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.79 to 1.0.80.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.79...v1.0.80 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-05-03 13:46:45 -06:00
554467ebd3
add no_os_memory_stats_reporting_arg ( #24938 )
2022-05-03 11:23:30 -05:00
8846aa12ca
ledger-tool: save new snapshots to --snapshot-archive-path ( #24883 )
2022-05-02 11:00:15 -06:00
428cf54c91
Change BlockStore TryPrimaryThenSecondary to just Secondary ( #23391 )
2022-04-29 20:05:39 -05:00
4e58b3870c
Update all BankForks methods to return owned values ( #24801 )
2022-04-28 18:51:00 +00:00
412a5a0d33
Credits auto rewind on vote recreation ( #22546 )
...
* Credits auto rewind on vote recreation
* Update comment
* Improve comments and tests
* Recommended fn rename
* Restore old feature, and replace new feature key
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2022-04-26 15:49:35 -06:00
533eadd024
Remove StakeState::get_rent_exempt_reserve()
2022-04-25 21:43:04 -07:00
83e041299a
Run real snapshot packager while processing blockstore at validator startup
2022-04-22 21:17:49 -07:00
2a770e9726
Use unsigned_abs() instead of casting abs() ( #24521 )
2022-04-20 12:01:33 -05:00
cfe2177e16
validator option: accounts-db-skip-rewrites ( #24504 )
2022-04-20 09:41:00 -05:00
9e4999ef6a
Remove halt_at_slot from RuntimeConfig, it's not a runtime concern
2022-04-19 19:23:58 -07:00
c6f3da4879
blockstore_processor now accepts an Arc<Rwlock<BankForks>>
2022-04-19 19:23:58 -07:00
aea17c35ae
Add a stringified credential option for LedgerStorage ( #24314 )
...
* add a stringified credential option for LedgerStorage
* fix clippy::useless-format warning
* change CredentialOption to enum CredentialType
* rename credential_option to credential_type
* restore LedgerStorage new fn signature
* fmt
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2022-04-13 14:35:06 -06:00
1bc49d219d
IndexLimitMb option adds 'Unspecified' state ( #24249 )
2022-04-12 09:38:09 -05:00
9b8850f99e
test-validator: Add `--max-compute-units` flag ( #24130 )
...
* test-validator: Add `--max-compute-units` flag
* Add `RuntimeConfig` for tweaking runtime behavior
* Actually add the file
* Move RuntimeConfig to runtime
2022-04-12 02:28:10 +02:00
60b2155bd3
Add accounts-filler-size command line option ( #23896 )
2022-04-11 13:10:09 -05:00
64abd008ca
make ledger-tool arg help consistent ( #24203 )
2022-04-08 15:45:09 -05:00
a058f348a2
Address review comments
2022-04-08 14:37:55 -05:00
c322842257
Replace channel with Mutex<Option> for AccountsPackage ( #24013 )
2022-04-06 05:47:19 -05:00
b61c0a4a21
Add accounts arg to genesis command to dump genesis account info ( #23879 )
2022-03-24 14:26:08 -05:00
c4ecfa5716
Bump version to v1.11 ( #23807 )
...
* Revert crossbeam_epoch to stable. 0.9.8 only works with nightly
* Remove unneeded unit expression
2022-03-21 17:40:50 -05:00
9b32b72990
bigtable: allow custom instance names
2022-03-21 11:26:49 -06:00
f513195468
bigtable: add a config ctor for `LedgerStorage`
2022-03-21 11:26:49 -06:00
f999eef452
(LedgerStore) Rename BlockstoreAdvancedOptions to LedgerColumnOptions ( #23764 )
...
This PR renames BlockstoreAdvancedOptions to LedgerColumnOptions, as we will
pass-down this struct to LedgerColumn to allow it to perform metric reporting.
2022-03-18 11:13:35 -07:00
2f58c9e501
Bump version to 1.10.4 ( #23743 )
2022-03-17 14:02:13 -05:00
0eccacbd5b
Add CLI support for versioned transactions ( #23606 )
2022-03-17 11:43:04 +08:00
2da4e3eb6c
Add --no-os-memory-stats-reporting
2022-03-15 17:07:40 -07:00
e7cf84a593
Remove AccessType arg from create_new_ledger ( #23591 )
...
Creating a new ledger implicitly means that no other process could have
previously held access to it. Additionally, creating a new ledger
implicitly requires writing, so it follows that Primary access is
required and we can drop access type as an argument.
2022-03-15 00:07:16 -05:00
8c8f9694e0
Refactor: Sanitized transaction creation ( #23558 )
...
* Refactor: SanitizedTransaction::try_create optionally computes hash
* Refactor: Add SimpleAddressLoader
2022-03-15 12:02:22 +08:00
115f376465
Factor out bank_forks_utils::load_bank_forks()
2022-03-14 15:29:58 -07:00
63bf0f66af
Bump version to 1.10.3 ( #23648 )
2022-03-14 11:18:45 -05:00
01b6f97f0b
Pin histogram crate version to fix cargo publish
2022-03-13 16:19:30 -07:00
1e20bd8f9a
(LedgerStore) Include storage type as a tag in RocksDB metric reporting ( #23523 )
...
#### Summary of Changes
This PR further enables group by operation on storage type in blockstore_rocksdb_cfs metrics.
Such group-by allows us to further compare the performance metrics between rocks-level and
rocks-fifo.
To make things extensible, this PR introduces BlockstoreAdvancedOptions and move shred_storage_type.
All fields in BlockstoreAdvancedOptions will support group-by operation in blockstore_rocksdb_cfs.
Dependency: #23580
2022-03-11 15:17:34 -08:00
d20dd21600
Sort tables in Cargo.toml files ( #23602 )
2022-03-11 11:05:57 -06:00
9b591286d7
Revert "chore: bump dashmap from 4.0.2 to 5.1.0 ( #23372 )" ( #23592 )
...
This reverts commit 3a0271c113
2022-03-11 09:12:27 +01:00
021135978d
Refactor: Split up cli transaction display methods ( #23547 )
2022-03-11 10:49:53 +08:00
b444836a97
Bump version to 1.10.2 ( #23597 )
2022-03-10 16:41:06 -06:00
3114c199bd
Add RPC support for versioned transactions ( #22530 )
...
* Add RPC support for versioned transactions
* fix doc tests
* Add rpc test for versioned txs
* Switch to preflight bank
2022-03-08 15:20:34 +08:00
3a0271c113
chore: bump dashmap from 4.0.2 to 5.1.0 ( #23372 )
...
* chore: bump dashmap from 4.0.2 to 5.1.0
Bumps [dashmap](https://github.com/xacrimon/dashmap ) from 4.0.2 to 5.1.0.
- [Release notes](https://github.com/xacrimon/dashmap/releases )
- [Commits](https://github.com/xacrimon/dashmap/commits/v5.1.0 )
---
updated-dependencies:
- dependency-name: dashmap
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-03-07 00:00:26 -05:00
61d7bdd66f
chore: bump serde_json from 1.0.78 to 1.0.79 ( #23461 )
...
* chore: bump serde_json from 1.0.78 to 1.0.79
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.78 to 1.0.79.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.78...v1.0.79 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2022-03-02 23:38:06 -07:00
634f4eb37d
(LedgerStore) Use different path for different blockstore storage type. ( #23236 )
...
#### Summary of Changes
To avoid mixing the use of different shred storage types, each shred storage type
will have its blockstore in a different directory.
This PR still keeps the RocksFifo setting hidden. The default ShredStorageType and
blockstore directory are still RocksLevel and `rocksdb`.
Will follow-up with PRs on making FIFO option public in ledger-tool and validator.
#### Test Plan
* Added a new test to verify the existence of `rocksdb-fifo` directory when FIFO compaction is used.
* Updated existing test to verify the current setting still store ledger under `rocksdb` directory.
* Manually ran ledger_cleanup_test with both level and fifo compaction and verified the resulting ledger.
* Ran a validator with this PR.
2022-03-02 18:30:22 -08:00
1a99251498
Bump version to 1.10.1 ( #23453 )
2022-03-02 13:47:01 -06:00
4bc440666a
add arg --disable_accounts_disk_index ( #23308 )
2022-02-23 18:07:24 -06:00
9213fcb11b
Introduce experimental FIFO-compaction option for shreds in blockstore ( #22140 )
2022-02-10 11:34:03 -08:00
d5dec989b9
Enforce tx metadata upload with static types ( #23028 )
2022-02-10 13:28:18 +08:00
60af1a4cce
Refactor: Add trait for loading addresses ( #22903 )
2022-02-03 11:00:12 +00:00
db481e1799
chore: bump serde_json from 1.0.75 to 1.0.78 ( #22748 )
...
* chore: bump serde_json from 1.0.75 to 1.0.78
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.75 to 1.0.78.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.75...v1.0.78 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-26 18:01:02 -07:00
95bbb70c91
chore: bump serde_json from 1.0.74 to 1.0.75 ( #22541 )
...
* chore: bump serde_json from 1.0.74 to 1.0.75
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.74 to 1.0.75.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.74...v1.0.75 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-21 12:26:51 -07:00
f804ccdece
Store address table lookups in blockstore and bigtable ( #22402 )
2022-01-14 15:24:41 +08:00
e291342c4a
Revert "chore: bump dashmap from 4.0.2 to 5.0.0 ( #21824 )" ( #22488 )
...
This reverts commit 8aa3d690b5
2022-01-13 13:06:39 -06:00
8b66625c95
convert std::sync::mpsc to crossbeam_channel ( #22264 )
2022-01-11 02:44:46 -08:00
5771c36d3f
Rename open_with_access_type() to open_with_options() ( #22123 )
2022-01-07 12:11:43 -08:00
9f1f64e384
Cleanup ledger-tool analyze-storage command ( #22310 )
...
* Make ledger-tool analyze-storage use Blockstore::open()
Opening a large ledger may require setting a larger open file descriptor
limit. Blockstore::open() does this whereas the underlying Database
object that analyze-storage was opening does not.
* Move key_size call lookup to take advantage of traits
* Fix typo where analyze worked on wrong column
* Make analyze-storage analyze all columns
2022-01-06 23:40:02 -06:00
d9220652ad
[ledger-tool]compare_blocks ( #22229 )
...
* 1.made load_credentials accept credential path as a parameter. 2.partial implement bigtable comparasion function
* finding missing blocks in bigtables in a specified range
* refactor compare-blocks,add unit test for missing_blocks and fmt
* compare-block fix last block bug
* refactor compare-block and improve wording
* Update ledger-tool/src/bigtable.rs
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* update compare-block command-line description
* style:improve wording/naming/code style
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2022-01-06 14:36:03 +08:00
e8b7f96a89
Add struct BlockstoreOptions ( #22121 )
2022-01-03 18:30:45 -10:00
ecbfc70bfa
Bump serde_json from 1.0.73 to 1.0.74 ( #22231 )
...
* Bump serde_json from 1.0.73 to 1.0.74
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.73 to 1.0.74.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.73...v1.0.74 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-03 10:36:59 -07:00
2a00382d71
Refactor: cleanup solana_transaction_status crate ( #22230 )
2022-01-03 15:45:18 +00:00
f67ecd5c18
removes unused Cargo dependencies ( #22022 )
...
Also moving some to [dev-dependencies] so that they are not propagated
to other packages which depend on the package.
2021-12-22 17:01:33 +00:00
3356c6afb2
chore: bump signal-hook from 0.3.12 to 0.3.13 ( #22037 )
...
Bumps [signal-hook](https://github.com/vorner/signal-hook ) from 0.3.12 to 0.3.13.
- [Release notes](https://github.com/vorner/signal-hook/releases )
- [Changelog](https://github.com/vorner/signal-hook/blob/master/CHANGELOG.md )
- [Commits](https://github.com/vorner/signal-hook/compare/v0.3.12...v0.3.13 )
---
updated-dependencies:
- dependency-name: signal-hook
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-20 21:40:16 -07:00
6ff0be6a82
Clean up demote program write lock feature ( #21949 )
...
* Clean up demote program write lock feature
* fix test
2021-12-16 17:27:22 -05:00
8d980f07ba
uses Option<Slot> for SlotMeta.parent_slot ( #21808 )
...
SlotMeta.parent_slot for the head of a detached chain of slots is
unknown and that is indicated by u64::MAX which lacks type-safety:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L203-L205
The commit changes the type to Option<Slot>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.
2021-12-14 18:57:11 +00:00
426f2507d5
chore: bump serde_yaml from 0.8.21 to 0.8.23 ( #21867 )
...
* chore: bump serde_yaml from 0.8.21 to 0.8.23
Bumps [serde_yaml](https://github.com/dtolnay/serde-yaml ) from 0.8.21 to 0.8.23.
- [Release notes](https://github.com/dtolnay/serde-yaml/releases )
- [Commits](https://github.com/dtolnay/serde-yaml/compare/0.8.21...0.8.23 )
---
updated-dependencies:
- dependency-name: serde_yaml
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-14 10:23:26 -07:00
018b54dbd7
chore: bump serde_json from 1.0.72 to 1.0.73 ( #21856 )
...
* chore: bump serde_json from 1.0.72 to 1.0.73
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.72 to 1.0.73.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.72...v1.0.73 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-14 04:08:22 +00:00
8aa3d690b5
chore: bump dashmap from 4.0.2 to 5.0.0 ( #21824 )
...
* chore: bump dashmap from 4.0.2 to 5.0.0
Bumps [dashmap](https://github.com/xacrimon/dashmap ) from 4.0.2 to 5.0.0.
- [Release notes](https://github.com/xacrimon/dashmap/releases )
- [Commits](https://github.com/xacrimon/dashmap/commits/v5.0.0 )
---
updated-dependencies:
- dependency-name: dashmap
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-13 13:23:58 -07:00
c965517a6b
chore: bump signal-hook from 0.3.10 to 0.3.12 ( #21721 )
...
Bumps [signal-hook](https://github.com/vorner/signal-hook ) from 0.3.10 to 0.3.12.
- [Release notes](https://github.com/vorner/signal-hook/releases )
- [Changelog](https://github.com/vorner/signal-hook/blob/master/CHANGELOG.md )
- [Commits](https://github.com/vorner/signal-hook/compare/v0.3.10...v0.3.12 )
---
updated-dependencies:
- dependency-name: signal-hook
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-09 21:43:31 +00:00
94b1cf47ca
Plumb verbose_level through ledger-tool slot subcommand ( #21670 )
...
* Add commas for readability and fix plurality in a printout
* Pass verbose_level into slot subcommand instead of default to max
This is useful for when a full printout isn't necessary, such as just
trying to determine the slot blockhash. The equivalent behavior before
this change would have been using "-vv" with the command.
2021-12-07 15:14:39 -06:00
f6801a4af4
chore: bump itertools from 0.10.1 to 0.10.3 ( #21643 )
...
* chore: bump itertools from 0.10.1 to 0.10.3
Bumps [itertools](https://github.com/rust-itertools/itertools ) from 0.10.1 to 0.10.3.
- [Release notes](https://github.com/rust-itertools/itertools/releases )
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.10.1...v0.10.3 )
---
updated-dependencies:
- dependency-name: itertools
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-06 23:23:26 -07:00
d6f22433d0
Bump version to v1.10.0
2021-12-04 20:17:54 +00:00
b8837c04ec
Reformat imports to a consistent style for imports
...
rustfmt.toml configuration:
imports_granularity = "One"
group_imports = "One"
2021-12-03 09:19:13 -08:00
2b269dbe0e
Ledger-tool: only require ledger dir when necessary ( #21575 )
...
* Don't canonicalize ledger_path unless ledger_path will be used
* Single use statement
2021-12-02 17:23:51 -07:00
dd12d90eac
Upgrade to Rust 2021
2021-11-30 20:43:46 -08:00
a125e7b8c2
chore: bump serde_json from 1.0.71 to 1.0.72 ( #21426 )
...
* chore: bump serde_json from 1.0.71 to 1.0.72
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.71 to 1.0.72.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.71...v1.0.72 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-30 18:03:09 +00:00
b1b334a17f
ledger-tool uses jemalloc like validator ( #21351 )
2021-11-19 09:48:07 -06:00
79d21d6805
add --accounts-index-scan-results-limit-mb to allow scans to abort ( #21327 )
...
* ScanConfig -> &ScanConfig
* add --accounts-index-scan-results-limit-mb to allow scans to abort
* feedback
2021-11-19 09:00:19 -06:00
89102540b1
chore: bump serde_json from 1.0.70 to 1.0.71 ( #21338 )
...
* chore: bump serde_json from 1.0.70 to 1.0.71
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.70 to 1.0.71.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.70...v1.0.71 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-18 02:58:30 +00:00
0f69a14247
Add ability to abort scan ( #21314 )
2021-11-17 13:10:29 -06:00
d5de0c8e12
add --no-os-network-stats-reporting option ( #21296 )
2021-11-16 10:26:03 -08:00
55ea672041
chore: bump serde_json from 1.0.69 to 1.0.70 ( #21285 )
...
* chore: bump serde_json from 1.0.69 to 1.0.70
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.69 to 1.0.70.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.69...v1.0.70 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-11-15 15:27:47 -07:00
77a59ca3e3
ledger-tool verify supports --accounts-db-skip-shrink ( #21284 )
2021-11-15 14:42:56 -06:00
a043b19988
ledger tool verify launches system monitor to report mem ( #21265 )
2021-11-15 09:29:30 -06:00
8b2133b078
Refactor total stats calculation into shared fn ( #21212 )
2021-11-09 19:13:37 -06:00
3c5f505d3e
chore: bump serde_json from 1.0.68 to 1.0.69 ( #21211 )
...
* chore: bump serde_json from 1.0.68 to 1.0.69
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.68 to 1.0.69.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.68...v1.0.69 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-09 11:04:40 -07:00
6f3c27f1da
Add total accounts stats to ledger-tool accounts ( #21187 )
2021-11-08 14:45:30 -06:00
2031522809
Add flags to not print account contents ( #21178 )
...
It takes a long time to print the contents of all the accounts. I'll be
adding more to the "accounts" subcommand and would like to skip printing
the account contents.
2021-11-05 08:59:12 -05:00
140a5f633d
Simplify replay vote tracking by using packet metadata ( #21112 )
2021-11-03 09:02:48 +00:00
855b3ccdc0
chore: bump signal-hook from 0.2.3 to 0.3.10 ( #21081 )
...
* chore: bump signal-hook from 0.2.3 to 0.3.10
Bumps [signal-hook](https://github.com/vorner/signal-hook ) from 0.2.3 to 0.3.10.
- [Release notes](https://github.com/vorner/signal-hook/releases )
- [Changelog](https://github.com/vorner/signal-hook/blob/master/CHANGELOG.md )
- [Commits](https://github.com/vorner/signal-hook/compare/v0.2.3...v0.3.10 )
---
updated-dependencies:
- dependency-name: signal-hook
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Update apis
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-10-30 00:21:54 -06:00
43ea579f63
add cli for --accounts-hash-num-passes ( #20827 )
2021-10-25 09:45:46 -05:00
c2bfce90b3
- cost_tracker is data member of a bank, it can report metrics when bank is frozen ( #20802 )
...
- removed cost_tracker_stats and histogram
- move stats reporting outside of bank freeze
2021-10-24 22:19:23 -05:00
350bb561eb
Clippy
2021-10-23 08:21:20 +00:00
50571685d4
Take incremental snapshots with ledger-tool ( #20857 )
2021-10-21 21:02:12 -05:00
b5f21d5e34
Add ledger tool to remove dead slot ( #20810 )
2021-10-19 20:23:16 -07:00
7496b5784b
- make cost_tracker a member of bank, remove shared instance from TPU; ( #20627 )
...
- decouple cost_model from cost_tracker; allowing one cost_model
instance being shared within a validator;
- update cost_model api to calculate_cost(&self...)->transaction_cost
2021-10-19 14:37:33 -05:00
3b40fa3f44
Add is_full to ledger-tool ( #20183 )
2021-10-13 12:50:18 +02:00
005d6863fd
- move cost tracker into bank, so each bank has its own cost tracker; ( #20527 )
...
- move related modules to runtime
2021-10-12 08:51:33 -05:00
a8e000a2a6
add filler accounts to bloat validator and predict failure ( #20491 )
...
* add filler accounts to bloat validator and predict failure
* assert no accounts match filler
* cleanup magic numbers
* panic if can't load from snapshot with filler accounts specified
* some renames
* renames
* into_par_iter
* clean filler accts, too
2021-10-11 12:46:27 -05:00
6cd1a03bb5
AcctIdx: fix ledger-tool arg options ( #20513 )
2021-10-07 19:43:21 +00:00
767f740305
Bump version to 1.9.0
2021-10-06 17:57:41 -07:00
7027d56064
Resolve nightly-2021-10-05 clippy complaints
2021-10-06 10:37:58 -07:00
03913f6661
add tx count and thread id to stats, each stat reports and resets when slot changes ( #20451 )
2021-10-06 00:09:19 -05:00
250a8503fe
Make rewards tracer async friendly ( #20452 )
2021-10-05 21:30:08 -04:00
6ff508c643
add transaction cost histogram metrics ( #20350 )
2021-10-05 08:57:39 -05:00
fe97cb2ddf
AccountsDb plugin framework ( #20047 )
...
Summary of Changes
Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows
Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.
The code comprises 4 major parts:
accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-09-30 14:26:17 -07:00
5810568c65
add --accounts-index-path to be used by disk accounts index ( #19710 )
...
* add --accounts-index-paths to be used by disk accounts index
* add multiple and default to help
2021-09-28 11:07:47 -05:00
22f45dca80
Make bigtable delete-slots actually usable ( #20037 )
2021-09-20 20:00:32 +00:00
c1d181add5
add --accounts-index-memory-limit-mb ( #19269 )
2021-09-19 18:00:15 -05:00
4e038e94fd
add 'drives' to AccountsIndexConfig ( #19989 )
2021-09-18 09:54:57 -05:00
99f5684dc4
AcctIdx: handle future config defaults ( #19967 )
2021-09-17 13:12:06 -05:00
c71fab6cb3
Add `delete` subcommand to `ledger-tool bigtable` ( #19931 )
...
* Add `delete` subcommand to `ledger-tool bigtable` command
* feedback
2021-09-16 23:37:45 +00:00
68da1909de
chore: bump serde_json from 1.0.67 to 1.0.68 ( #19942 )
...
* chore: bump serde_json from 1.0.67 to 1.0.68
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.67 to 1.0.68.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.67...v1.0.68 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-16 20:23:25 +00:00
4c69229a47
Add incremental snapshot CLI args to ledger-tool ( #19889 )
...
This commit only adds args to specify the maximum full/incremental
snapshot archives to retain when purging old snapshot archives. It
purposely does *not* add a way to generate new incremental snapshots.
Fixes #19857
2021-09-16 10:56:18 -05:00
b57e86abf2
cache account hash info ( #19426 )
...
* cache account hash info
* ledger_path -> accounts_hash_cache_path
2021-09-13 20:39:26 -05:00
36f870a7e9
chore: bump serde_yaml from 0.8.20 to 0.8.21 ( #19820 )
...
* chore: bump serde_yaml from 0.8.20 to 0.8.21
Bumps [serde_yaml](https://github.com/dtolnay/serde-yaml ) from 0.8.20 to 0.8.21.
- [Release notes](https://github.com/dtolnay/serde-yaml/releases )
- [Commits](https://github.com/dtolnay/serde-yaml/compare/0.8.20...0.8.21 )
---
updated-dependencies:
- dependency-name: serde_yaml
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-13 15:50:33 +00:00
62c8bcf565
Add default() to SnapshotConfig ( #19776 )
2021-09-12 13:44:27 -05:00
0f76077969
Add `--destake-vote-account <VOTE_ADDRESS>...` argument to `create-snapshot` command ( #19749 )
2021-09-11 14:44:37 -07:00
c3a610374a
unused deps
2021-09-11 14:33:53 -07:00
456bf15012
AccountsIndexConfig -> AccountsDbConfig ( #19687 )
2021-09-08 04:30:38 +00:00
a0552e5b46
Make startup aware of Incremental Snapshots ( #19600 )
2021-09-07 20:43:43 +00:00
9d9482b9d8
Plumb `maximum_incremental_snapshot_archives_to_retain` ( #19640 )
2021-09-06 18:01:56 -05:00
decec3cd8b
Demote write locks on transaction program ids ( #19593 )
...
* Add feature
* Demote write lock on program ids
* Fixup bpf tests
* Update MappedMessage::is_writable
* Comma nit
* Review comments
2021-09-04 03:05:30 +00:00
5e25ee5ebe
Add maximum_incremental_snapshot_archives_to_retain to SnapshotConfig ( #19612 )
2021-09-03 20:21:32 +00:00
7ab0aec61f
Rename maximum_full_snapshot_archives_to_retain ( #19610 )
...
To prepare for adding maximum_incremental_snapshot_archives_to_retain,
rename the current field in SnapshotConfig.
2021-09-03 11:28:10 -05:00
e9374d32a3
Revert "Make startup aware of Incremental Snapshots ( #19550 )" ( #19599 )
...
This reverts commit d45ced0a5d
2021-09-02 19:14:41 -05:00
d45ced0a5d
Make startup aware of Incremental Snapshots ( #19550 )
2021-09-02 19:05:15 -05:00
1d5a8ebc6a
Revert "Add LastFullSnapshotSlot to SnapshotConfig ( #19341 )" ( #19529 )
...
This reverts commit 4d361af976
2021-08-31 22:03:19 -05:00
fe9ee9134a
Make background services aware of incremental snapshots ( #19401 )
...
AccountsBackgroundService now knows about incremental snapshots. It is
now also in charge of deciding if an AccountsPackage is destined to be a
SnapshotPackage or not (or just used by AccountsHashVerifier).
!!! New behavior changes !!!
Taking snapshots (both bank and archive) **MUST** succeed.
This is required because of how the last full snapshot slot is
calculated, which is used by AccountsBackgroundService when calling
`clean_accounts()`.
File system calls are now unwrapped and will result in a crash. As Trent told me:
>Well I think if a snapshot fails due to some IO error, it's very likely that the operator is going to have to intervene before it works. We should exit error in this case, otherwise the validator might happily spin for several more hours, never successfully writing a complete snapshot, before something else brings it down. This would leave the validator's last local snapshot many more slots behind than it would be had we exited outright and potentially force the operator to abandon ledger continuity in favor of a quick catchup
Other errors will set the `exit` flag to `true`, and the node will gracefully shutdown.
Fixes #19167
Fixes #19168
2021-08-31 18:33:27 -05:00
c21bf29ce7
chore: bump serde_json from 1.0.66 to 1.0.67 ( #19501 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.66 to 1.0.67.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.66...v1.0.67 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-31 06:05:25 +00:00
8e4360cf9c
chore: bump futures from 0.3.16 to 0.3.17 ( #19513 )
...
* chore: bump futures from 0.3.16 to 0.3.17
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.16 to 0.3.17.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.16...0.3.17 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-08-30 22:12:38 -06:00
fbffdf9f3d
chore: bump futures-util from 0.3.16 to 0.3.17 ( #19498 )
...
* chore: bump futures-util from 0.3.16 to 0.3.17
Bumps [futures-util](https://github.com/rust-lang/futures-rs ) from 0.3.16 to 0.3.17.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.16...0.3.17 )
---
updated-dependencies:
- dependency-name: futures-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-30 17:45:47 +00:00
8ad52fa095
implements copy-on-write for vote-accounts ( #19362 )
...
Bank::vote_accounts redundantly clones vote-accounts HashMap even though
an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/95c998a19/runtime/src/bank.rs#L5174-L5186
This commit implements copy-on-write semantics for vote-accounts by
wrapping the underlying HashMap in Arc<...>.
2021-08-30 15:54:01 +00:00
db2be00289
chore: bump serde_yaml from 0.8.19 to 0.8.20 ( #19464 )
...
* chore: bump serde_yaml from 0.8.19 to 0.8.20
Bumps [serde_yaml](https://github.com/dtolnay/serde-yaml ) from 0.8.19 to 0.8.20.
- [Release notes](https://github.com/dtolnay/serde-yaml/releases )
- [Commits](https://github.com/dtolnay/serde-yaml/compare/0.8.19...0.8.20 )
---
updated-dependencies:
- dependency-name: serde_yaml
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-27 09:33:08 -06:00
c721300ab5
chore: bump serde_yaml from 0.8.18 to 0.8.19 ( #19372 )
...
* chore: bump serde_yaml from 0.8.18 to 0.8.19
Bumps [serde_yaml](https://github.com/dtolnay/serde-yaml ) from 0.8.18 to 0.8.19.
- [Release notes](https://github.com/dtolnay/serde-yaml/releases )
- [Commits](https://github.com/dtolnay/serde-yaml/compare/0.8.18...0.8.19 )
---
updated-dependencies:
- dependency-name: serde_yaml
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-08-23 10:23:16 -06:00
6d939811e9
Name snapshots consistently ( #19346 )
...
#### Problem
Snapshot names are overloaded, and there are multiple terms that mean the same thing. This is confusing. Here's a list of ones in the codebase that I've found:
```
- snapshot_dir
- snapshots_dir
- snapshot_path
- snapshot_output_dir
- snapshot_package_output_path
- snapshot_archives_dir
```
#### Summary of Changes
For all the ones that are about the directory where snapshot archives are stored, ensure they are `snapshot_archives_dir`. For the ones about the (bank) snapshots directory, set to `bank_snapshots_dir`.
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-08-21 15:41:03 -05:00
4d361af976
Add LastFullSnapshotSlot to SnapshotConfig ( #19341 )
2021-08-20 17:06:53 +00:00
Yueh-Hsuan Chiang
Will Hickey
Jeff Washington (jwash)
Tyera Eulberg
Xiang Zhu
Brooks Prumo
Jeff Washington (jwash)
dependabot[bot]
apfitzge
steviez
Richard Patel
github-actions[bot]
Brennan Watt
Trent Nelson
Nicholas Rempel
Dan Coombs
Jeff Biseda
kirill lykov
Michael Vines
sakridge
HaoranYi
DimAn
Justin Starry
Ryo Onodera
Rachael Pai
Jon Cinque
Giorgio Gambino
Christian Kamm
behzad nouri
Tyera Eulberg
Jeff Washington (jwash)
Tao Zhu
carllin
Lijun Wang
Giles Cope
Brooks Prumo