* Add `ProgramCosts` Column Family to blockstore, implement LedgerColumn; add `delete_cf` to Rocks
* Add ProgramCosts to compaction excluding list alone side with TransactionStatusIndex in one place: `excludes_from_compaction()`
* Write cost table to blockstore after `replay_stage` replayed active banks; add stats to measure persist time
* Deletes program from `ProgramCosts` in blockstore when they are removed from cost_table in memory
* Only try to persist to blockstore when cost_table is changed.
* Restore cost table during validator startup
* Offload `cost_model` related operations from replay main thread to dedicated service thread, add channel to send execute_timings between these threads;
* Move `cost_update_service` to its own module; replay_stage is now decoupled from cost_model.
* Update rocksdb to v0.16.0
* Promote the infrequent and important log to info!
* Force background compaction by ttl without manual compaction
* Fix test
* Support no compaction mode in test_ledger_cleanup_compaction
* Fix comment
* Make compaction_interval customizable
* Avoid major compaction with periodic filtering...
* Adress lazy_static, special cfs and range check
* Clean up a bit and add comment
* Add comment
* More comments...
* Config code cleanup
* Add comment
* Use .conflicts_with()
* Nullify unneeded delete_range ops for special CFs
* Some clean ups
* Clarify the locking intention
* Ensure special CFs' consistency with PurgeType::CompactionFilter
* Fix comment
* Fix bad copy paste
* Fix various types...
* Don't use tuples
* Add a unit test for compaction_filter
* Fix typo...
* Remove flag and just use new behavior always
* Fix wrong condition negation...
* Doc. about no set_last_purged_slot in purge_slots
* Write a test and fix off-by-one bug....
* Apply suggestions from code review
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Follow up to github review suggestions
* Fix line-wrapping
* Fix conflict
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Add BlockHeight CF to blockstore
* Rename CacheBlockTimeService to be more general
* Cache block-height using service
* Fixup previous proto mishandling
* Add block_height to block structs
* Add block-height to solana block
* Fallback to BankForks if block time or block height are not yet written to Blockstore
* Add docs
* Review comments
* Add blockstore-root-scan for api nodes on boot
* Ensure cluster-confirmed root and parents are set as root in blockstore in load_frozen_forks()
* Plumb rpc-scan-and-fix-roots validator flag
* Upgrade Rust to 1.52.0
update nightly_version to newly pushed docker image
fix clippy lint errors
1.52 comes with grcov 0.8.0, include this version to script
* upgrade to Rust 1.52.1
* disabling Serum from downstream projects until it is upgraded to Rust 1.52.1
* Require that blockstore block-time only be recognized slot, instead of root
* Move cache_block_time to after Bank freeze
* Single use statement
* Pass transaction_status_sender by reference
* Remove unnecessary slot-existence check before caching block time altogether
* Move block-time existence check into Blockstore::cache_block_time, Blockstore no longer needed in blockstore_processor helper
CodingShredHeader.position is equal to
ShredCommonHeader.index - ShredCommonHeader.fec_set_index
and is so redundant. The extra position field can add bugs if not
consistent with index and fec_set_index.
Strip the zero-padding off of data shreds before insertion into blockstore
Co-authored-by: Stephen Akridge <sakridge@gmail.com>
Co-authored-by: Nathan Hawkins <utsl@utsl.org>
Number of parity coding shreds is always less than the number of data
shreds in FEC blocks:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L719
Data shreds are batched in chunks of 32 shreds each:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L714
However the very last batch of data shreds in a slot can be small, in
which case the loss rate can be exacerbated.
This commit expands the number of coding shreds in the last FEC block in
slots to: 64 - number of data shreds; so that FEC blocks are always 64
data and parity coding shreds each.
As a consequence of this, the last FEC block has more parity coding
shreds than data shreds. So for some shred indices we will have a coding
shred but no data shreds. This should not cause any kind of overlapping
FEC blocks as in:
https://github.com/solana-labs/solana/pull/10095
since this is done only for the very last batch in a slot, and the next
slot will reset the shred index.
* Update blockstore method to allow return of unfinalized signature
* Support confirmed sigs in getConfirmedSignaturesForAddress2
* Add deprecated comments
* Update docs
* Enable confirmed transaction-history in cli
* Return real confirmation_status; fill in not-yet-finalized block time if possible
* add block_time to get_confirmed_signatures_for_address2 and protobuf implementation for tx_by_addr
* add tests for convert
* update cargo lock
* run cargo format after rebase
* introduce legacy TransactionByAddrInfo
* move LegacyTransactionByAddrInfo back to storage-bigtable
* feat: store pre / post token balances
* move helper functions into separate include
* move token balance functionality to transaction-status crate
* fix blockstore processor test
* fix bigtable legacy test
* add caching to decimals
Gossip and other places repeatedly de-serialize vote-state stored in
vote accounts. Ideally the first de-serialization should cache the
result.
This commit adds new VoteAccount type which lazily de-serializes
VoteState from Account data and caches the result internally.
Serialize and Deserialize traits are manually implemented to match
existing code. So, despite changes to frozen_abi, this commit should be
backward compatible.
* Add Blockstore protobuf cf type
* Add Rewards message to proto and make generated pub
* Convert Rewards cf to ProtobufColumn
* Add bench
* Adjust tags
* Move solana proto definitions and conversion methods to new crate
* Move timestamp helper to sdk
* Add Bank method for getting timestamp estimate
* Return sysvar info from Bank::clock
* Add feature-gated timestamp correction
* Rename unix_timestamp method to be more descriptive
* Review comments
* Add timestamp metric
* introduce store program logs in blockstore / bigtable
* fix test, transaction logs created for successful transactions
* fix test for legacy bincode implementation around log_messages
* only api nodes should record logs
* truncate transaction logs to 100KB
* refactor log truncate for improved coverage
* Add blockstore column to store performance sampling data
* introduce timer and write performance metrics to blockstore
* introduce getRecentPerformanceSamples rpc
* only run on rpc nodes enabled with transaction history
* add unit tests for get_recent_performance_samples
* remove RpcResponse from rpc call
* refactor to use Instant::now and elapsed for timer
* switch to root bank and ensure not negative subraction
* Add PerfSamples to purge/compaction
* refactor to use Instant::now and elapsed for timer
* switch to root bank and ensure not negative subraction
* remove duplicate constants
Co-authored-by: Tyera Eulberg <tyera@solana.com>
* Add blockstore column to cache block times
* Add method to cache block time
* Add service to cache block time
* Update rpc getBlockTime to use new method, and refactor blockstore slightly
* Return block_time with confirmed block, if available
* Add measure and warning to cache-block-time
* Submit a timestamp for every vote
* Submit at most one vote timestamp per second
* Submit a timestamp for every new vote
Co-authored-by: Tyera Eulberg <tyera@solana.com>
* Refactor bigtable apis to accept start and end keys
* Make helper fn to deserialize cell data
* Refactor get_confirmed_signatures_for_address to use get_row_data range
* Add until param to get_confirmed_signatures_for_address
* Add until param to blockstore api
* Plumb until through client/cli
* Simplify client params
* Freeze address-signature index in the middle of slot to show failure case
* Secondary filter on signature
* Use AddressSignatures iterator instead of manually decrementing slots
* Remove unused method
* Add metrics
* Add transaction-status-index doccumentation
* Add panicking test
* Add failing test: fresh transaction-status column shouldn't point at valid root 0
* Prevent transaction status match outside of primary-index bounds
* Initialize transaction-status and address-signature primer entries with Slot::MAX
* Revert "Add failing test: fresh transaction-status column shouldn't point at valid root 0"
This reverts commit cbad2a9fae22e5531e3b4ff1b0a9d6a223826c71.
* Revert "Initialize transaction-status and address-signature primer entries with Slot::MAX"
This reverts commit ffaeac0669d0cbe18dd68b5ce177e15a92360b72.
* Make Message::new_with_payer the default constructor
* Remove Transaction::new_[un]signed_instructions
These guess the fee-payer instead of stating it explicitly
* Revert "Untar is called for shred archives that do not exist. (#9565)"
This reverts commit 729cb5eec6.
* Revert "Dont insert shred payload into rocksdb (#9366)"
This reverts commit 5ed39de8c5.
* Separate client types into own crate, so ledger does not need it
Removes about 50 crates of dependency from ledger
* Drop Rpc name from transaction-status types
* Split timestamp calculation into separate fn for math unit testing
* Add failing test
* Fix failing test; also bump stakes to near expected cluster max supply
* Don't error on timestamp of slot 0
* Consolidate entry tick verification into one function
* Mark bad slots as dead in blocktree processor
* more feedback
* Add bank.is_complete
* feedback
carllin
Michael Vines
Tao Zhu
Lijun Wang
sakridge
Alexander Meißner
Tyera Eulberg
steviez
Ryo Onodera
behzad nouri
François Garillot
Justin Starry
Trent Nelson
Josh
carllin
Tyera Eulberg
Greg Fitzgerald
Kristofer Peterson
Ryo Onodera
anatoly yakovenko
Jack May
Pankaj Garg
Sagar Dhawan