b8b7163b66
(Ledger Store) Report RocksDB Column Family Metrics ( #22503 )
...
This PR enables blockstore to periodically report RocksDB column family properties.
The reported properties are under blockstore_rocksdb_cfs, and the properties also
support group by operation on cf_name.
2022-03-05 16:13:03 -08:00
36ad59673c
drop mut
2022-03-04 09:52:46 +01:00
0d33b54d74
Rework do_process_blockstore_from_root to use BankForks
2022-03-04 09:52:46 +01:00
93c8e04d51
Simplify do_process_blockstore_from_root slightly
2022-03-04 09:52:46 +01:00
360f6466a3
chore: bump futures from 0.3.19 to 0.3.21 ( #23476 )
...
* chore: bump futures from 0.3.19 to 0.3.21
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.19 to 0.3.21.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.19...0.3.21 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-03-03 16:44:43 -07:00
62d2a4cd88
Make ShredStorageType::RocksLevel public ( #23272 )
...
#### Summary of Changes
This PR adds two hidden arguments to the validator that allow users to use RocksDB's FIFO compaction for storing shreds.
--shred-storage <SHRED_STORAGE>
EXPERIMENTAL: Controls how RocksDB compacts shreds. *WARNING*: You will lose your ledger data
when you switch between options. Possible values are: 'level': stores shreds using RocksDB's default (level)
compaction. 'fifo': stores shreds under RocksDB's FIFO compaction. This option is more efficient on
disk-write-bytes of the ledger store. [default: level] [possible values: level, fifo]
--shred-storage-size <SHRED_STORAGE_SIZE_BYTES>
The shred storage size in bytes. The suggested value is 50% of your ledger storage size in bytes. [default:
268435456000]
2022-03-03 12:43:58 -08:00
634f4eb37d
(LedgerStore) Use different path for different blockstore storage type. ( #23236 )
...
#### Summary of Changes
To avoid mixing the use of different shred storage types, each shred storage type
will have its blockstore in a different directory.
This PR still keeps the RocksFifo setting hidden. The default ShredStorageType and
blockstore directory are still RocksLevel and `rocksdb`.
Will follow-up with PRs on making FIFO option public in ledger-tool and validator.
#### Test Plan
* Added a new test to verify the existence of `rocksdb-fifo` directory when FIFO compaction is used.
* Updated existing test to verify the current setting still store ledger under `rocksdb` directory.
* Manually ran ledger_cleanup_test with both level and fifo compaction and verified the resulting ledger.
* Ran a validator with this PR.
2022-03-02 18:30:22 -08:00
1a99251498
Bump version to 1.10.1 ( #23453 )
2022-03-02 13:47:01 -06:00
ef8b7d9c62
ledger tool halt at slot verify hash ( #23424 )
2022-03-02 11:11:18 -06:00
3b5b71ce44
Improve UX querying rpc for blocks at or before the snapshot from boot ( #23403 )
...
* Bump first-available block to first complete block
* Remove obsolete purges in tests (PrimaryIndex toggling no longer in use
* Check first-available block in Rpc check_slot_cleaned_up
2022-02-28 23:57:41 -07:00
985af71241
chore: bump sha2 from 0.10.1 to 0.10.2 ( #23343 )
...
* chore: bump sha2 from 0.10.1 to 0.10.2
Bumps [sha2](https://github.com/RustCrypto/hashes ) from 0.10.1 to 0.10.2.
- [Release notes](https://github.com/RustCrypto/hashes/releases )
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.10.1...sha2-v0.10.2 )
---
updated-dependencies:
- dependency-name: sha2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-02-24 23:57:42 -07:00
39c86c6c44
fmt ( #23329 )
2022-02-24 12:12:29 -06:00
12bddbc4ec
Use rocksdb property constant instead of local constant ( #23147 )
2022-02-23 23:02:11 -06:00
d0e85c293f
Fix rustfmt check ( #23296 )
2022-02-23 16:38:53 +08:00
20d031e2b8
Refactor ExecuteTimings w/ enum-indexed array ( #23085 )
2022-02-22 14:46:56 -08:00
8c872e9ce0
(LedgerStore/FIFO) Refactor FIFO options and sanity check. ( #23131 )
2022-02-19 00:58:38 -08:00
1719d2349f
Skip adding builtins if they will be removed ( #23233 )
...
* Add failing test for precompile transition
* Skip adding builtins if they will be removed
* cargo clean
* nits
* fix abi check
* remove workaround
Co-authored-by: Jack May <jack@solana.com>
2022-02-18 18:36:59 -08:00
7939fdc3e5
Minor refactor on Shreds column family descriptor construction. ( #23103 )
2022-02-16 00:29:11 -08:00
c04438be4b
Retaining transaction logs when transaction plugin is loaded. ( #22874 )
...
Transaction logs are not being saved to the database through the plugin interface.
Summary of Changes
Retain the transaction logs when transaction notification plugin is loaded.
Fixes #
lijunwangs/solana-accountsdb-plugin-postgres#6
2022-02-11 20:29:07 -08:00
6399647e64
(Ledger Store) Add code comments for SlotMetaWorkingSetEntry. ( #22662 )
2022-02-11 10:04:26 -08:00
e4a1799334
fix typos in ledge code comments ( #23068 )
2022-02-10 20:43:44 -07:00
9213fcb11b
Introduce experimental FIFO-compaction option for shreds in blockstore ( #22140 )
2022-02-10 11:34:03 -08:00
d5dec989b9
Enforce tx metadata upload with static types ( #23028 )
2022-02-10 13:28:18 +08:00
2ca7ae8e30
Update rocksdb to 0.18.0 release ( #23031 )
2022-02-09 21:51:56 -06:00
9d036be61b
chore: bump libc from 0.2.115 to 0.2.117 ( #23034 )
...
* chore: bump libc from 0.2.115 to 0.2.117
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.115 to 0.2.117.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.115...0.2.117 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-02-09 16:49:44 -07:00
a146f2d853
suppress clippy
2022-02-07 08:02:58 -08:00
c5d8560cdb
updates to address review feedback
2022-02-07 08:02:58 -08:00
dfef68f985
fix build
2022-02-07 08:02:58 -08:00
4de14e530b
Optimize batching of transactions during replay for parallel processing
2022-02-07 08:02:58 -08:00
37afdd1a65
(Ledger Store) Improve comments for blockstore_purge ( #22808 )
2022-02-06 21:56:23 -08:00
c7ca2f41f5
Add some comments for RocksDB option configuration ( #22655 )
2022-02-06 23:28:24 -06:00
ba215e94f6
Refactor: Add AccountKeys struct for static and dynamic message keys ( #22960 )
2022-02-05 20:00:31 +08:00
fb95fa68a2
cleanup api ( #22845 )
2022-01-31 11:39:23 -06:00
aaae5b3ba6
(Ledger Store) Add comment blocks for six pub functions in blockstore.rs ( #22476 )
2022-01-28 22:25:07 -08:00
40a49081b9
(Ledger Store) Add comments for find slot functions. ( #22666 )
2022-01-28 12:07:09 -08:00
cc74693176
chore: bump serde from 1.0.134 to 1.0.136 ( #22817 )
...
* chore: bump serde from 1.0.134 to 1.0.136
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.134 to 1.0.136.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.134...v1.0.136 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-28 12:20:37 -07:00
7376efe8ea
chore: bump libc from 0.2.112 to 0.2.115 ( #22796 )
...
* chore: bump libc from 0.2.112 to 0.2.115
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.112 to 0.2.115.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.112...0.2.115 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-27 21:54:13 +00:00
6d5bbca630
Pacify clippy
2022-01-21 19:12:57 -08:00
0cf886302d
chore: bump serde from 1.0.133 to 1.0.134 ( #22650 )
...
* chore: bump serde from 1.0.133 to 1.0.134
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.133 to 1.0.134.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.133...v1.0.134 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-21 17:14:02 -07:00
fe7543c31a
(Ledger Store) APIs for obtaining physical size of all data and coding shreds ( #22443 )
2022-01-19 19:31:19 -08:00
a724fa2347
Add hidden cli option to allow validator reports replayed transaction cost metrics ( #22369 )
...
* add hidden cli option to allow validator reports replayed transaction cost detail metrics
* Update validator/src/main.rs
Co-authored-by: Michael Vines <mvines@gmail.com>
* - rebase master, using unbounded instead of channel; dowgrade to datapoint_trace
* removed cli arg, prefer log at trace
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-01-15 00:31:21 +00:00
f804ccdece
Store address table lookups in blockstore and bigtable ( #22402 )
2022-01-14 15:24:41 +08:00
eca8d21249
log internals ( #22493 )
2022-01-13 19:50:46 -05:00
8b66625c95
convert std::sync::mpsc to crossbeam_channel ( #22264 )
2022-01-11 02:44:46 -08:00
28275a33d6
Bump tempfile from 3.2.0 to 3.3.0 ( #22401 )
...
* Bump tempfile from 3.2.0 to 3.3.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases )
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS )
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.2.0...v3.3.0 )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-09 12:26:08 -07:00
2759cb860b
Bump sha2 from 0.10.0 to 0.10.1 ( #22359 )
...
* Bump sha2 from 0.10.0 to 0.10.1
Bumps [sha2](https://github.com/RustCrypto/hashes ) from 0.10.0 to 0.10.1.
- [Release notes](https://github.com/RustCrypto/hashes/releases )
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.10.0...sha2-v0.10.1 )
---
updated-dependencies:
- dependency-name: sha2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-08 00:17:32 -07:00
813006b33b
remove per program timings from blockstore processor ledger replay ( #22370 )
2022-01-08 01:52:08 -05:00
5771c36d3f
Rename open_with_access_type() to open_with_options() ( #22123 )
2022-01-07 12:11:43 -08:00
207825d30b
Minimize boilerplate code around Rocks column families ( #22345 )
2022-01-06 23:39:09 -06:00
52d12cc802
Add runtime support for address table lookups ( #22223 )
...
* Add support for address table lookups in runtime
* feedback
* feedback
2022-01-07 11:59:09 +08:00
Yueh-Hsuan Chiang
Michael Vines
dependabot[bot]
Will Hickey
Jeff Washington (jwash)
Tyera Eulberg
steviez
Justin Starry
Gavin Chan
Lijun Wang
HaoranYi
Pankaj Garg
Tao Zhu
carllin
Jeff Biseda