blockworks-foundation
/
solana
mirror of https://github.com/blockworks-foundation/solana.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
22,253 Commits 168 Branches 497 Tags 539 MiB
Commit Graph

653 Commits

Author SHA1 Message Date
Richard Patel f7c6901191
ledger-tool: support Geyser accounts updates (#26909) 2022-08-09 15:44:57 +00:00
dependabot[bot] 07a0076fc2
chore: bump bytecount from 0.6.2 to 0.6.3 (#27014) 
Bumps [bytecount](https://github.com/llogiq/bytecount) from 0.6.2 to 0.6.3.
- [Release notes](https://github.com/llogiq/bytecount/releases)
- [Commits](https://github.com/llogiq/bytecount/commits)

---
updated-dependencies:
- dependency-name: bytecount
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-08-09 00:33:43 -06:00
Will Hickey ed8c224374
Bump version to v1.12 (#26967) 2022-08-06 13:20:30 -05:00
Tyera Eulberg 66919e5adb
Unpin tokio for non-rpc crates (#26957) 2022-08-05 20:50:45 -06:00
steviez 300666dce7
Make `solana-ledger-tool` run AccountsBackgroundService (#26914) 
Prior to this change, long running commands like `solana-ledger-tool
verify` would OOM due to AccountsDb cleanup not happening.

Co-authored-by: Michael Vines <mvines@gmail.com>
 2022-08-04 15:44:31 -05:00
github-actions[bot] fbf1bf6d86
Bump Version to 1.11.6 (#26906) 
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
 2022-08-03 12:48:43 -05:00
Brennan Watt f3b760dd91
Add IO metrics (#26804) 
* Add Disk IO metrics
 2022-08-02 14:29:53 -07:00
Jeff Washington (jwash) c7462b7a52
ledger tool verify can store debug info on hash calc (#26837) 2022-07-29 15:54:56 -05:00
github-actions[bot] 5d038b9d2a
Bump Version to 1.11.5 (#26758) 
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
 2022-07-25 13:05:14 -06:00
Trent Nelson 0fff5b8bf8 ledger-tool: optionally `graph` vote states 2022-07-22 17:43:01 -06:00
Trent Nelson 633950e029 ledger-tool: factor out `graph` options to config struct 2022-07-22 17:43:01 -06:00
Trent Nelson a541827fb9 ledger-tool: fix off-by-one in `graph` skipped slot label 2022-07-22 17:43:01 -06:00
Yueh-Hsuan Chiang dcbda2c6c5
Allow ledger tool to automatically detect the shred compaction style (#26182) 
#### Problem
Ledger-tool doesn't support shred-compaction-type other than the default rocksdb level compaction.

#### Summary of Changes
This PR enables ledger-tool to automatically detect the shred-compaction-type of the specified ledger.

#### Test Plan
New ledger-tool tests are added for both level and fifo compactions.
 2022-07-19 01:19:17 +08:00
github-actions[bot] fd5df1cf25
Bump Version to 1.11.4 (#26578) 
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
 2022-07-11 23:30:38 -05:00
apfitzge 1c2f6a4c41
Bugfix: slots_connected snapshot_slot not full (#26506) 
* slots_connected check used by ledger-tool should not require a full slot for snapshot slot

* Cleaner Result<Option<>> unwrap/default

* return false if no meta for starting slot

* Add clarifying comments
 2022-07-11 13:45:12 -05:00
github-actions[bot] 9d937fb8a0
Bump Version to 1.11.3 (#26481) 
Co-authored-by: willhickey <willhickey@users.noreply.github.com>
 2022-07-07 14:39:46 -05:00
Jeff Washington (jwash) 26176af4aa
cleanup cli arg help (#26366) 2022-07-05 14:49:05 -05:00
Jeff Washington (jwash) 95ae82e074
add info to ledger tool for total execution time (#26368) 2022-07-05 09:23:06 -05:00
Nicholas Rempel 2b21c1da29 Sort Cargo.toml files 2022-06-24 12:41:38 -07:00
github-actions[bot] 5c2f819f99
Bump Version to 1.11.2 (#26159) 2022-06-22 21:16:18 -05:00
apfitzge f4189c0305
ledger-tool minimized snapshots (#25334) 
* working on local snapshot

* Parallelization for slot storage minimization

* Additional clean-up and fixes

* make --minimize an option of create-snapshot

* remove now unnecessary function

* Parallelize parts of minimized account set generation

* clippy fixes

* Add rent collection accounts and voting node_pubkeys

* Simplify programdata_accounts generation

* Loop over storages to get slot set

* Parallelize minimized slot set generation

* Parallelize adding owners and programdata_accounts

* Remove some now unncessary checks on the blockstore

* Add a warning for minimized snapshots across epoch boundary

* Simplify ledger-tool minimize

* Clarify names of bank's minimization helper functions

* Remove unnecesary funciton, fix line spacing

* Use DashSets instead of HashSets for minimized account and slot sets

* Filter storages uses all threads instead of thread_pool

* Add some additional comments on functions for minimization

* Moved more into bank and parallelized

* Update programs/bpf/Cargo.lock for dashmap in ledger

* Clippy fix

* ledger-tool: convert minimize_bank_for_snapshot Measure into measure!

* bank.rs: convert minimize_bank_for_snapshot Measure into measure!

* accounts_db.rs: convert minimize_accounts_db Measure into measure!

* accounts_db.rs: add comment about use of minimize_accounts_db

* ledger-tool: CLI argument clarification

* minimization functions: make infos unique

* bank.rs: Add test_get_rent_collection_accounts_between_slots

* bank.rs: Add test_minimization_add_vote_accounts

* bank.rs: Add test_minimization_add_stake_accounts

* bank.rs: Add test_minimization_add_owner_accounts

* bank.rs: Add test_minimization_add_programdata_accounts

* accounts_db.rs: Add test_minimize_accounts_db

* bank.rs: Add negative case and comments in test_get_rent_collection_accounts_between_slots

* bank.rs: Negative test in test_minimization_add_programdata_accounts

* use new static runtime and sdk ids

* bank comments to doc comments

* Only need to insert the maximum slot a key is found in

* rename remove_pubkeys to purge_pubkeys

* add comment on builtins::get_pubkeys

* prevent excessive logging of removed dead slots

* don't need to remove slot from shrink slot candidates

* blockstore.rs: get_accounts_used_in_range shouldn't return Result

* blockstore.rs: get_accounts_used_in_range: parallelize slot loop

* report filtering progress on time instead of count

* parallelize loop over snapshot storages

* WIP: move some bank minimization functionality into a new class

* WIP: move some accounts_db minimization functionality into SnapshotMinimizer

* WIP: Use new SnapshotMinimizer

* SnapshotMinimizer: fix use statements

* remove bank and accounts_db minimization code, where possible

* measure! doesn't take a closure

* fix use statement in blockstore

* log_dead_slots does not need pub(crate)

* get_unique_accounts_from_storages does not need pub(crate)

* different way to get stake accounts/nodes

* fix tests

* move rent collection account functionality to snapshot minimizer

* move accounts_db minimize behavior to snapshot minimizer

* clean up

* Use bank reference instead of Arc. Additional comments

* Add a comment to blockstore function

* Additional clarifying comments

* Moved all non-transaction account accumulation into the SnapshotMinimizer.

* transaction_account_set does not need to be mutable now

* Add comment about load_to_collect_rent_eagerly

* Update log_dead_slots comment

* remove duplicate measure/print of get_minimized_slot_set
 2022-06-22 13:17:43 -04:00
Jeff Washington (jwash) 6513bb3b1b
add options to create snapshot (#26095) 2022-06-21 13:07:05 -05:00
Will Hickey 51f26dc96e
Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
Tyera Eulberg 2866ca4b1c
Add ledger-tool bigtable upload loop (#26030) 
* Add ledger-tool bigtable upload loop

* Limit range on caller side, switch to while loop, and remove now-obsolete option
 2022-06-17 19:31:13 +00:00
Tyera Eulberg a9c9acfe80
Improve repair-roots help (#26008) 2022-06-16 10:03:27 -06:00
Dan Coombs 32a58dd9e0
Add ability to use a non-default app profile id in bigtable requests (#25968) 
* Add ability to use a non-default app profile id in bigtable requests

* Only run subcommand once when getting global configs

* Remove unneded scoping on option type
 2022-06-16 00:58:16 +00:00
Brennan Watt ba04063956
Add CPUmetrics (#25802) 
Add in some CPU utilization metrics such as: number of vCPUs, clock frequency, average load across different time intervals, and number of total threads
 2022-06-07 11:34:25 -07:00
apfitzge 934da5ef99
Fix pre-check of blockstore slots during load_bank_forks (#25632) 
Fix pre-check of blockstore slts during load_bank_forks. Now iterates from starting_slot to halt_slot via slot_meta.next_slots to confirm they are connected.
 2022-06-01 20:19:42 -05:00
Yueh-Hsuan Chiang 5b67960c76
(Refactor) Move blocktore options related stuff to blockstore_options.rs (#25509) 
#### Problem
blockstore_db.rs has a mutual dependency between blockstore_metrics.rs.

#### Summary of Changes
This PR removes the mutual dependency by moving the option-related stuff
out from blockstore_db.rs to its new home --- blockstore_options.rs.

By doing this, we address the mutual dependency and also make the code cleaner.
 2022-05-26 16:59:26 -07:00
apfitzge 328cca9fcd
Add a --debug-key option to ledger-tool verify (#25391) 2022-05-24 19:56:35 -05:00
Jeff Biseda 61c5a471e8
preserve optimistic_slot in blockstore (#25311) 2022-05-24 12:03:28 -07:00
kirill lykov 2cb500a388
fix problem with matches in ledger-tool (#25482) 2022-05-23 12:39:58 -05:00
Michael Vines 9d9773bd2a Use write! instead of format! to pacify clippy 2022-05-22 22:22:21 -07:00
Michael Vines b05c7d91ed Fix derive_partial_eq_without_eq clippy lint 2022-05-22 22:22:21 -07:00
steviez d65aab4d27
Fix ledger-tool slot to output tx even when tx status is missing (#25434) 2022-05-20 20:56:06 -06:00
Jeff Washington (jwash) 467431de89
add ledger-tool accounts-db-skip-initial-hash-calc (#25354) 2022-05-20 10:27:00 -05:00
apfitzge 1e7d877494
Create-snapshot check if slot is available on startup (#25329) 
* Add a check during ledger-tool create-snapshot startup to see if the snapshot slot is available

* check all slots from the start to snapshot_slot during load_bank_forks

* unwrap_or_default incremental snapshot slot before comparison

* Improve error messages on missing or not full slots
 2022-05-20 09:11:18 -05:00
sakridge cdac141417
Sort per-program and print a summary of all slots in the range (#25267) 2022-05-19 23:26:29 +02:00
apfitzge 83b245642f
--snapshot-archive-format should not conflict w/ --no-snapshot (#25368) 2022-05-19 12:49:59 -05:00
dependabot[bot] 3b47250d4a
chore: bump signal-hook from 0.3.13 to 0.3.14 (#25281) 
* chore: bump signal-hook from 0.3.13 to 0.3.14

Bumps [signal-hook](https://github.com/vorner/signal-hook) from 0.3.13 to 0.3.14.
- [Release notes](https://github.com/vorner/signal-hook/releases)
- [Changelog](https://github.com/vorner/signal-hook/blob/master/CHANGELOG.md)
- [Commits](https://github.com/vorner/signal-hook/compare/v0.3.13...v0.3.14)

---
updated-dependencies:
- dependency-name: signal-hook
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-05-18 09:28:52 -06:00
steviez 3d8e5e91ed
Fix ledger-tool from erroring with unspecified archive_format (#25298) 
value_t_or_exit()! will error if the user doesn't specify a value at
runtime, use value_of() instead which will give either the default value
or whatever the user specified.
 2022-05-17 15:46:31 -06:00
HaoranYi 3e44046a73
Support lz4 for snapshot archives (#25089) 
* add lz4

* add cargo package file

* fix tests

* use fast lz4 compression

* report snapshot archive format

* add test

* code review feedback

* add cargo.lock

* fix var name

* refactor archive format parsing and add default compress cli arg

* clippy

* add from_cli_arg test

* update cargo.lock

* add lz4 support for mmap

* cargo.lock

* clippy
 2022-05-16 12:44:15 -05:00
Tyera Eulberg bc005e3408
Add configurable limit to number of blocks to check before Bigtable upload (#24716) 
* Add ConfirmedBlockUploadConfig, no behavior changes

* Add comment

* A little DRY cleanup

* Add configurable limit to number of blocks to check in Blockstore and Bigtable before uploading

* Limit blockstore and bigtable look-ahead

* Exit iterator early when reach ending_slot

* Use rooted_slot_iterator instead of slot_meta_iterator

* Only check blocks in the ledger
 2022-05-13 07:34:02 +00:00
Jeff Washington (jwash) a8930ee14b
add --accounts-db-ancient-append-vecs (#25125) 2022-05-11 08:47:07 -05:00
DimAn 2fa9bc3e70
Add options to store full and/or incremental snapshots in separate locations (#24247) 2022-05-10 16:37:41 -04:00
apfitzge b83fd8b958
ledger-tool: clear non-primary access accounts directory in load_bank_forks (#24958) 2022-05-09 13:17:13 -05:00
Justin Starry 082502d4f3
Fail tx sanitization when ix program id uses lookup table (#25035) 
* Fail tx sanitization when ix program id uses lookup table

* feedback
 2022-05-07 03:19:50 +08:00
Tyera Eulberg c4aa082efc
Rollback tokio to LTS release v1.14 (#25028) 2022-05-06 06:34:49 +00:00
dependabot[bot] bece7f32c8
chore: bump log from 0.4.16 to 0.4.17 (#24987) 
* chore: bump log from 0.4.16 to 0.4.17

Bumps [log](https://github.com/rust-lang/log) from 0.4.16 to 0.4.17.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/commits/0.4.17)

---
updated-dependencies:
- dependency-name: log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-05-05 01:49:18 +00:00
dependabot[bot] 61a9faae17
chore: bump serde_json from 1.0.80 to 1.0.81 (#24960) 
* chore: bump serde_json from 1.0.80 to 1.0.81

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.80 to 1.0.81.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.80...v1.0.81)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2022-05-04 14:57:38 -06:00
dependabot[bot] 2c9d2a2140
chore: bump serde_json from 1.0.79 to 1.0.80 (#24943) 
* chore: bump serde_json from 1.0.79 to 1.0.80

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.79 to 1.0.80.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.79...v1.0.80)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-05-03 13:46:45 -06:00
HaoranYi 554467ebd3
add no_os_memory_stats_reporting_arg (#24938) 2022-05-03 11:23:30 -05:00
DimAn 8846aa12ca
ledger-tool: save new snapshots to --snapshot-archive-path (#24883) 2022-05-02 11:00:15 -06:00
steviez 428cf54c91
Change BlockStore TryPrimaryThenSecondary to just Secondary (#23391) 2022-04-29 20:05:39 -05:00
Justin Starry 4e58b3870c
Update all BankForks methods to return owned values (#24801) 2022-04-28 18:51:00 +00:00
Ryo Onodera 412a5a0d33
Credits auto rewind on vote recreation (#22546) 
* Credits auto rewind on vote recreation

* Update comment

* Improve comments and tests

* Recommended fn rename

* Restore old feature, and replace new feature key

Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2022-04-26 15:49:35 -06:00
Michael Vines 533eadd024 Remove StakeState::get_rent_exempt_reserve() 2022-04-25 21:43:04 -07:00
Michael Vines 83e041299a Run real snapshot packager while processing blockstore at validator startup 2022-04-22 21:17:49 -07:00
steviez 2a770e9726
Use unsigned_abs() instead of casting abs() (#24521) 2022-04-20 12:01:33 -05:00
Jeff Washington (jwash) cfe2177e16
validator option: accounts-db-skip-rewrites (#24504) 2022-04-20 09:41:00 -05:00
Michael Vines 9e4999ef6a Remove halt_at_slot from RuntimeConfig, it's not a runtime concern 2022-04-19 19:23:58 -07:00
Michael Vines c6f3da4879 blockstore_processor now accepts an Arc<Rwlock<BankForks>> 2022-04-19 19:23:58 -07:00
Rachael Pai aea17c35ae
Add a stringified credential option for LedgerStorage (#24314) 
* add a stringified credential option for LedgerStorage

* fix clippy::useless-format warning

* change CredentialOption to enum CredentialType

* rename credential_option to credential_type

* restore LedgerStorage new fn signature

* fmt

Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2022-04-13 14:35:06 -06:00
Jeff Washington (jwash) 1bc49d219d
IndexLimitMb option adds 'Unspecified' state (#24249) 2022-04-12 09:38:09 -05:00
Jon Cinque 9b8850f99e
test-validator: Add `--max-compute-units` flag (#24130) 
* test-validator: Add `--max-compute-units` flag

* Add `RuntimeConfig` for tweaking runtime behavior

* Actually add the file

* Move RuntimeConfig to runtime
 2022-04-12 02:28:10 +02:00
Giorgio Gambino 60b2155bd3
Add accounts-filler-size command line option (#23896) 2022-04-11 13:10:09 -05:00
Jeff Washington (jwash) 64abd008ca
make ledger-tool arg help consistent (#24203) 2022-04-08 15:45:09 -05:00
Christian Kamm a058f348a2 Address review comments 2022-04-08 14:37:55 -05:00
Brooks Prumo c322842257
Replace channel with Mutex<Option> for AccountsPackage (#24013) 2022-04-06 05:47:19 -05:00
steviez b61c0a4a21
Add accounts arg to genesis command to dump genesis account info (#23879) 2022-03-24 14:26:08 -05:00
Will Hickey c4ecfa5716
Bump version to v1.11 (#23807) 
* Revert crossbeam_epoch to stable. 0.9.8 only works with nightly
* Remove unneeded unit expression
 2022-03-21 17:40:50 -05:00
Trent Nelson 9b32b72990 bigtable: allow custom instance names 2022-03-21 11:26:49 -06:00
Trent Nelson f513195468 bigtable: add a config ctor for `LedgerStorage` 2022-03-21 11:26:49 -06:00
Yueh-Hsuan Chiang f999eef452
(LedgerStore) Rename BlockstoreAdvancedOptions to LedgerColumnOptions (#23764) 
This PR renames BlockstoreAdvancedOptions to LedgerColumnOptions, as we will
pass-down this struct to LedgerColumn to allow it to perform metric reporting.
 2022-03-18 11:13:35 -07:00
Will Hickey 2f58c9e501
Bump version to 1.10.4 (#23743) 2022-03-17 14:02:13 -05:00
Justin Starry 0eccacbd5b
Add CLI support for versioned transactions (#23606) 2022-03-17 11:43:04 +08:00
Michael Vines 2da4e3eb6c Add --no-os-memory-stats-reporting 2022-03-15 17:07:40 -07:00
steviez e7cf84a593
Remove AccessType arg from create_new_ledger (#23591) 
Creating a new ledger implicitly means that no other process could have
previously held access to it. Additionally, creating a new ledger
implicitly requires writing, so it follows that Primary access is
required and we can drop access type as an argument.
 2022-03-15 00:07:16 -05:00
Justin Starry 8c8f9694e0
Refactor: Sanitized transaction creation (#23558) 
* Refactor: SanitizedTransaction::try_create optionally computes hash

* Refactor: Add SimpleAddressLoader
 2022-03-15 12:02:22 +08:00
Michael Vines 115f376465 Factor out bank_forks_utils::load_bank_forks() 2022-03-14 15:29:58 -07:00
Will Hickey 63bf0f66af
Bump version to 1.10.3 (#23648) 2022-03-14 11:18:45 -05:00
Michael Vines 01b6f97f0b Pin histogram crate version to fix cargo publish 2022-03-13 16:19:30 -07:00
Yueh-Hsuan Chiang 1e20bd8f9a
(LedgerStore) Include storage type as a tag in RocksDB metric reporting (#23523) 
#### Summary of Changes
This PR further enables group by operation on storage type in blockstore_rocksdb_cfs metrics.
Such group-by allows us to further compare the performance metrics between rocks-level and
rocks-fifo.

To make things extensible, this PR introduces BlockstoreAdvancedOptions and move shred_storage_type. 
All fields in BlockstoreAdvancedOptions will support group-by operation in blockstore_rocksdb_cfs.

Dependency: #23580
 2022-03-11 15:17:34 -08:00
Brooks Prumo d20dd21600
Sort tables in Cargo.toml files (#23602) 2022-03-11 11:05:57 -06:00
sakridge 9b591286d7
Revert "chore: bump dashmap from 4.0.2 to 5.1.0 (#23372)" (#23592) 
This reverts commit 3a0271c113.
 2022-03-11 09:12:27 +01:00
Justin Starry 021135978d
Refactor: Split up cli transaction display methods (#23547) 2022-03-11 10:49:53 +08:00
Will Hickey b444836a97
Bump version to 1.10.2 (#23597) 2022-03-10 16:41:06 -06:00
Justin Starry 3114c199bd
Add RPC support for versioned transactions (#22530) 
* Add RPC support for versioned transactions

* fix doc tests

* Add rpc test for versioned txs

* Switch to preflight bank
 2022-03-08 15:20:34 +08:00
dependabot[bot] 3a0271c113
chore: bump dashmap from 4.0.2 to 5.1.0 (#23372) 
* chore: bump dashmap from 4.0.2 to 5.1.0

Bumps [dashmap](https://github.com/xacrimon/dashmap) from 4.0.2 to 5.1.0.
- [Release notes](https://github.com/xacrimon/dashmap/releases)
- [Commits](https://github.com/xacrimon/dashmap/commits/v5.1.0)

---
updated-dependencies:
- dependency-name: dashmap
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-03-07 00:00:26 -05:00
dependabot[bot] 61d7bdd66f
chore: bump serde_json from 1.0.78 to 1.0.79 (#23461) 
* chore: bump serde_json from 1.0.78 to 1.0.79

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.78 to 1.0.79.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.78...v1.0.79)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2022-03-02 23:38:06 -07:00
Yueh-Hsuan Chiang 634f4eb37d
(LedgerStore) Use different path for different blockstore storage type. (#23236) 
#### Summary of Changes
To avoid mixing the use of different shred storage types, each shred storage type
will have its blockstore in a different directory.

This PR still keeps the RocksFifo setting hidden.  The default ShredStorageType and
blockstore directory are still RocksLevel and `rocksdb`.

Will follow-up with PRs on making FIFO option public in ledger-tool and validator.

#### Test Plan
* Added a new test to verify the existence of `rocksdb-fifo` directory when FIFO compaction is used.
* Updated existing test to verify the current setting still store ledger under `rocksdb` directory.
* Manually ran ledger_cleanup_test with both level and fifo compaction and verified the resulting ledger.
* Ran a validator with this PR.
 2022-03-02 18:30:22 -08:00
Will Hickey 1a99251498
Bump version to 1.10.1 (#23453) 2022-03-02 13:47:01 -06:00
Jeff Washington (jwash) 4bc440666a
add arg --disable_accounts_disk_index (#23308) 2022-02-23 18:07:24 -06:00
Yueh-Hsuan Chiang 9213fcb11b
Introduce experimental FIFO-compaction option for shreds in blockstore (#22140) 2022-02-10 11:34:03 -08:00
Justin Starry d5dec989b9
Enforce tx metadata upload with static types (#23028) 2022-02-10 13:28:18 +08:00
Justin Starry 60af1a4cce
Refactor: Add trait for loading addresses (#22903) 2022-02-03 11:00:12 +00:00
dependabot[bot] db481e1799
chore: bump serde_json from 1.0.75 to 1.0.78 (#22748) 
* chore: bump serde_json from 1.0.75 to 1.0.78

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.75 to 1.0.78.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.75...v1.0.78)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-01-26 18:01:02 -07:00
dependabot[bot] 95bbb70c91
chore: bump serde_json from 1.0.74 to 1.0.75 (#22541) 
* chore: bump serde_json from 1.0.74 to 1.0.75

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.74 to 1.0.75.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.74...v1.0.75)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-01-21 12:26:51 -07:00
Justin Starry f804ccdece
Store address table lookups in blockstore and bigtable (#22402) 2022-01-14 15:24:41 +08:00
Jeff Washington (jwash) e291342c4a
Revert "chore: bump dashmap from 4.0.2 to 5.0.0 (#21824)" (#22488) 
This reverts commit 8aa3d690b5.
 2022-01-13 13:06:39 -06:00
Jeff Biseda 8b66625c95
convert std::sync::mpsc to crossbeam_channel (#22264) 2022-01-11 02:44:46 -08:00
Yueh-Hsuan Chiang 5771c36d3f
Rename open_with_access_type() to open_with_options() (#22123) 2022-01-07 12:11:43 -08:00
steviez 9f1f64e384
Cleanup ledger-tool analyze-storage command (#22310) 
* Make ledger-tool analyze-storage use Blockstore::open()

Opening a large ledger may require setting a larger open file descriptor
limit. Blockstore::open() does this whereas the underlying Database
object that analyze-storage was opening does not.

* Move key_size call lookup to take advantage of traits

* Fix typo where analyze worked on wrong column

* Make analyze-storage analyze all columns
 2022-01-06 23:40:02 -06:00
pieceofr d9220652ad
[ledger-tool]compare_blocks (#22229) 
* 1.made load_credentials accept credential path as a parameter. 2.partial implement bigtable comparasion function

* finding missing blocks in bigtables in a specified range

* refactor compare-blocks,add unit test for missing_blocks and fmt

* compare-block fix last block bug

* refactor compare-block and improve wording

* Update ledger-tool/src/bigtable.rs

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* update compare-block command-line description

* style:improve wording/naming/code style

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
 2022-01-06 14:36:03 +08:00
Yueh-Hsuan Chiang e8b7f96a89
Add struct BlockstoreOptions (#22121) 2022-01-03 18:30:45 -10:00
dependabot[bot] ecbfc70bfa
Bump serde_json from 1.0.73 to 1.0.74 (#22231) 
* Bump serde_json from 1.0.73 to 1.0.74

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.73 to 1.0.74.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.73...v1.0.74)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-01-03 10:36:59 -07:00
Justin Starry 2a00382d71
Refactor: cleanup solana_transaction_status crate (#22230) 2022-01-03 15:45:18 +00:00
behzad nouri f67ecd5c18
removes unused Cargo dependencies (#22022) 
Also moving some to [dev-dependencies] so that they are not propagated
to other packages which depend on the package.
 2021-12-22 17:01:33 +00:00
dependabot[bot] 3356c6afb2
chore: bump signal-hook from 0.3.12 to 0.3.13 (#22037) 
Bumps [signal-hook](https://github.com/vorner/signal-hook) from 0.3.12 to 0.3.13.
- [Release notes](https://github.com/vorner/signal-hook/releases)
- [Changelog](https://github.com/vorner/signal-hook/blob/master/CHANGELOG.md)
- [Commits](https://github.com/vorner/signal-hook/compare/v0.3.12...v0.3.13)

---
updated-dependencies:
- dependency-name: signal-hook
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-20 21:40:16 -07:00
Justin Starry 6ff0be6a82
Clean up demote program write lock feature (#21949) 
* Clean up demote program write lock feature

* fix test
 2021-12-16 17:27:22 -05:00
behzad nouri 8d980f07ba
uses Option<Slot> for SlotMeta.parent_slot (#21808) 
SlotMeta.parent_slot for the head of a detached chain of slots is
unknown and that is indicated by u64::MAX which lacks type-safety:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L203-L205

The commit changes the type to Option<Slot>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.
 2021-12-14 18:57:11 +00:00
dependabot[bot] 426f2507d5
chore: bump serde_yaml from 0.8.21 to 0.8.23 (#21867) 
* chore: bump serde_yaml from 0.8.21 to 0.8.23

Bumps [serde_yaml](https://github.com/dtolnay/serde-yaml) from 0.8.21 to 0.8.23.
- [Release notes](https://github.com/dtolnay/serde-yaml/releases)
- [Commits](https://github.com/dtolnay/serde-yaml/compare/0.8.21...0.8.23)

---
updated-dependencies:
- dependency-name: serde_yaml
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-12-14 10:23:26 -07:00
dependabot[bot] 018b54dbd7
chore: bump serde_json from 1.0.72 to 1.0.73 (#21856) 
* chore: bump serde_json from 1.0.72 to 1.0.73

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.72 to 1.0.73.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.72...v1.0.73)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-12-14 04:08:22 +00:00
dependabot[bot] 8aa3d690b5
chore: bump dashmap from 4.0.2 to 5.0.0 (#21824) 
* chore: bump dashmap from 4.0.2 to 5.0.0

Bumps [dashmap](https://github.com/xacrimon/dashmap) from 4.0.2 to 5.0.0.
- [Release notes](https://github.com/xacrimon/dashmap/releases)
- [Commits](https://github.com/xacrimon/dashmap/commits/v5.0.0)

---
updated-dependencies:
- dependency-name: dashmap
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-12-13 13:23:58 -07:00
dependabot[bot] c965517a6b
chore: bump signal-hook from 0.3.10 to 0.3.12 (#21721) 
Bumps [signal-hook](https://github.com/vorner/signal-hook) from 0.3.10 to 0.3.12.
- [Release notes](https://github.com/vorner/signal-hook/releases)
- [Changelog](https://github.com/vorner/signal-hook/blob/master/CHANGELOG.md)
- [Commits](https://github.com/vorner/signal-hook/compare/v0.3.10...v0.3.12)

---
updated-dependencies:
- dependency-name: signal-hook
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-09 21:43:31 +00:00
steviez 94b1cf47ca
Plumb verbose_level through ledger-tool slot subcommand (#21670) 
* Add commas for readability and fix plurality in a printout
* Pass verbose_level into slot subcommand instead of default to max

This is useful for when a full printout isn't necessary, such as just
trying to determine the slot blockhash. The equivalent behavior before
this change would have been using "-vv" with the command.
 2021-12-07 15:14:39 -06:00
dependabot[bot] f6801a4af4
chore: bump itertools from 0.10.1 to 0.10.3 (#21643) 
* chore: bump itertools from 0.10.1 to 0.10.3

Bumps [itertools](https://github.com/rust-itertools/itertools) from 0.10.1 to 0.10.3.
- [Release notes](https://github.com/rust-itertools/itertools/releases)
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.10.1...v0.10.3)

---
updated-dependencies:
- dependency-name: itertools
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-12-06 23:23:26 -07:00
Trent Nelson d6f22433d0 Bump version to v1.10.0 2021-12-04 20:17:54 +00:00
Michael Vines b8837c04ec Reformat imports to a consistent style for imports 
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
 2021-12-03 09:19:13 -08:00
Tyera Eulberg 2b269dbe0e
Ledger-tool: only require ledger dir when necessary (#21575) 
* Don't canonicalize ledger_path unless ledger_path will be used

* Single use statement
 2021-12-02 17:23:51 -07:00
Michael Vines dd12d90eac Upgrade to Rust 2021 2021-11-30 20:43:46 -08:00
dependabot[bot] a125e7b8c2
chore: bump serde_json from 1.0.71 to 1.0.72 (#21426) 
* chore: bump serde_json from 1.0.71 to 1.0.72

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.71 to 1.0.72.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.71...v1.0.72)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-11-30 18:03:09 +00:00
Jeff Washington (jwash) b1b334a17f
ledger-tool uses jemalloc like validator (#21351) 2021-11-19 09:48:07 -06:00
Jeff Washington (jwash) 79d21d6805
add --accounts-index-scan-results-limit-mb to allow scans to abort (#21327) 
* ScanConfig -> &ScanConfig

* add --accounts-index-scan-results-limit-mb to allow scans to abort

* feedback
 2021-11-19 09:00:19 -06:00
dependabot[bot] 89102540b1
chore: bump serde_json from 1.0.70 to 1.0.71 (#21338) 
* chore: bump serde_json from 1.0.70 to 1.0.71

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.70 to 1.0.71.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.70...v1.0.71)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-11-18 02:58:30 +00:00
Jeff Washington (jwash) 0f69a14247
Add ability to abort scan (#21314) 2021-11-17 13:10:29 -06:00
Jeff Biseda d5de0c8e12
add --no-os-network-stats-reporting option (#21296) 2021-11-16 10:26:03 -08:00
dependabot[bot] 55ea672041
chore: bump serde_json from 1.0.69 to 1.0.70 (#21285) 
* chore: bump serde_json from 1.0.69 to 1.0.70

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.69 to 1.0.70.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.69...v1.0.70)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-11-15 15:27:47 -07:00
Jeff Washington (jwash) 77a59ca3e3
ledger-tool verify supports --accounts-db-skip-shrink (#21284) 2021-11-15 14:42:56 -06:00
Jeff Washington (jwash) a043b19988
ledger tool verify launches system monitor to report mem (#21265) 2021-11-15 09:29:30 -06:00
Brooks Prumo 8b2133b078
Refactor total stats calculation into shared fn (#21212) 2021-11-09 19:13:37 -06:00
dependabot[bot] 3c5f505d3e
chore: bump serde_json from 1.0.68 to 1.0.69 (#21211) 
* chore: bump serde_json from 1.0.68 to 1.0.69

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.68 to 1.0.69.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.68...v1.0.69)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-11-09 11:04:40 -07:00
Brooks Prumo 6f3c27f1da
Add total accounts stats to ledger-tool accounts (#21187) 2021-11-08 14:45:30 -06:00
Brooks Prumo 2031522809
Add flags to not print account contents (#21178) 
It takes a long time to print the contents of all the accounts.  I'll be
adding more to the "accounts" subcommand and would like to skip printing
the account contents.
 2021-11-05 08:59:12 -05:00
Justin Starry 140a5f633d
Simplify replay vote tracking by using packet metadata (#21112) 2021-11-03 09:02:48 +00:00
dependabot[bot] 855b3ccdc0
chore: bump signal-hook from 0.2.3 to 0.3.10 (#21081) 
* chore: bump signal-hook from 0.2.3 to 0.3.10

Bumps [signal-hook](https://github.com/vorner/signal-hook) from 0.2.3 to 0.3.10.
- [Release notes](https://github.com/vorner/signal-hook/releases)
- [Changelog](https://github.com/vorner/signal-hook/blob/master/CHANGELOG.md)
- [Commits](https://github.com/vorner/signal-hook/compare/v0.2.3...v0.3.10)

---
updated-dependencies:
- dependency-name: signal-hook
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update apis

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-10-30 00:21:54 -06:00
Jeff Washington (jwash) 43ea579f63
add cli for --accounts-hash-num-passes (#20827) 2021-10-25 09:45:46 -05:00
Tao Zhu c2bfce90b3
- cost_tracker is data member of a bank, it can report metrics when bank is frozen (#20802) 
- removed cost_tracker_stats and histogram
- move stats reporting outside of bank freeze
 2021-10-24 22:19:23 -05:00
Michael Vines 350bb561eb Clippy 2021-10-23 08:21:20 +00:00
Brooks Prumo 50571685d4
Take incremental snapshots with ledger-tool (#20857) 2021-10-21 21:02:12 -05:00
carllin b5f21d5e34
Add ledger tool to remove dead slot (#20810) 2021-10-19 20:23:16 -07:00
Tao Zhu 7496b5784b
- make cost_tracker a member of bank, remove shared instance from TPU; (#20627) 
- decouple cost_model from cost_tracker; allowing one cost_model
  instance being shared within a validator;
- update cost_model api to calculate_cost(&self...)->transaction_cost
 2021-10-19 14:37:33 -05:00
sakridge 3b40fa3f44
Add is_full to ledger-tool (#20183) 2021-10-13 12:50:18 +02:00
Tao Zhu 005d6863fd
- move cost tracker into bank, so each bank has its own cost tracker; (#20527) 
- move related modules to runtime
 2021-10-12 08:51:33 -05:00
Jeff Washington (jwash) a8e000a2a6
add filler accounts to bloat validator and predict failure (#20491) 
* add filler accounts to bloat validator and predict failure

* assert no accounts match filler

* cleanup magic numbers

* panic if can't load from snapshot with filler accounts specified

* some renames

* renames

* into_par_iter

* clean filler accts, too
 2021-10-11 12:46:27 -05:00
Jeff Washington (jwash) 6cd1a03bb5
AcctIdx: fix ledger-tool arg options (#20513) 2021-10-07 19:43:21 +00:00
Trent Nelson 767f740305 Bump version to 1.9.0 2021-10-06 17:57:41 -07:00
Michael Vines 7027d56064 Resolve nightly-2021-10-05 clippy complaints 2021-10-06 10:37:58 -07:00
Tao Zhu 03913f6661
add tx count and thread id to stats, each stat reports and resets when slot changes (#20451) 2021-10-06 00:09:19 -05:00
Justin Starry 250a8503fe
Make rewards tracer async friendly (#20452) 2021-10-05 21:30:08 -04:00