The manual Blockstore compaction that was being initiated from
LedgerCleanupService has been disabled for quite some time in favor of
several optimizations.
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
#### Problem
RocksDB's delete_range applies to [from, to) while delete_file_in_range
applies to [from, to] by default, and the rust-rocksdb api does not include
the option to make delete_file_in_range apply to [from, to). Such inconsistency
might cause `blockstore::run_purge` to produce an inconsistent result as it
invokes both delete_range and delete_file_in_range.
#### Summary of Changes
This PR makes all our purge / delete related functions to be inclusive
on both starting and ending slots.
#### Problem
LedgerCleanupService requires compactions to propagate & digest range-delete tombstones
to eventually reclaim disk space.
#### Summary of Changes
This PR makes LedgerCleanupService::cleanup_ledger delete any file whose slot-range is
older than the lowest_cleanup_slot. This allows us to reclaim disk space more often with
fewer IOps. Experimental results on mainnet validators show that the PR can effectively
reduce 33% to 40% ledger disk size.
* Fix link target in doc comment
* Fix formatting of log examples in process_instruction
* Fix doc markdown in solana-gossip
* Fix doc markdown in solana-runtime
* Escape square braces in doc comments to avoid warnings
* Surround 'account references' doc items in code spans to avoid warnings
* Fix code block in loader_upgradeable_instruction
* Fix doctest for loader_upgradable_instruction
* Update rocksdb to v0.16.0
* Promote the infrequent and important log to info!
* Force background compaction by ttl without manual compaction
* Fix test
* Support no compaction mode in test_ledger_cleanup_compaction
* Fix comment
* Make compaction_interval customizable
* Avoid major compaction with periodic filtering...
* Adress lazy_static, special cfs and range check
* Clean up a bit and add comment
* Add comment
* More comments...
* Config code cleanup
* Add comment
* Use .conflicts_with()
* Nullify unneeded delete_range ops for special CFs
* Some clean ups
* Clarify the locking intention
* Ensure special CFs' consistency with PurgeType::CompactionFilter
* Fix comment
* Fix bad copy paste
* Fix various types...
* Don't use tuples
* Add a unit test for compaction_filter
* Fix typo...
* Remove flag and just use new behavior always
* Fix wrong condition negation...
* Doc. about no set_last_purged_slot in purge_slots
* Write a test and fix off-by-one bug....
* Apply suggestions from code review
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Follow up to github review suggestions
* Fix line-wrapping
* Fix conflict
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Upgrade Rust to 1.52.0
update nightly_version to newly pushed docker image
fix clippy lint errors
1.52 comes with grcov 0.8.0, include this version to script
* upgrade to Rust 1.52.1
* disabling Serum from downstream projects until it is upgraded to Rust 1.52.1
* Add blockstore column to store performance sampling data
* introduce timer and write performance metrics to blockstore
* introduce getRecentPerformanceSamples rpc
* only run on rpc nodes enabled with transaction history
* add unit tests for get_recent_performance_samples
* remove RpcResponse from rpc call
* refactor to use Instant::now and elapsed for timer
* switch to root bank and ensure not negative subraction
* Add PerfSamples to purge/compaction
* refactor to use Instant::now and elapsed for timer
* switch to root bank and ensure not negative subraction
* remove duplicate constants
Co-authored-by: Tyera Eulberg <tyera@solana.com>
* Add blockstore column to cache block times
* Add method to cache block time
* Add service to cache block time
* Update rpc getBlockTime to use new method, and refactor blockstore slightly
* Return block_time with confirmed block, if available
* Add measure and warning to cache-block-time
* Freeze address-signature index in the middle of slot to show failure case
* Secondary filter on signature
* Use AddressSignatures iterator instead of manually decrementing slots
* Remove unused method
* Add metrics
* Add transaction-status-index doccumentation
steviez
behzad nouri
Yueh-Hsuan Chiang
Jeff Biseda
HaoranYi
Justin Starry
Michael Vines
Brian Anderson
sakridge
carllin
Ryo Onodera
Tao Zhu
Tyera Eulberg
Josh