* Fix bug where ReplayStage holds an Arc<Bank> for process lifetime
When ReplayStage::new() kicks off, it needs to do some setup with the
working bank prior to entering the main processing loop. This setup is
done before entering the main processing loop; however, a bug made it
such that an Arc<Bank> remained in scope after the processing loop had
been entered. The processing loop is only exited when the process exits,
so this means that Bank was being held for the lifetime of the process.
This is a waste of resources and prevents background cleanup.
* clippy
Extracted time metrics related to transaction execution into a separate
structure. This allows me to call `process_entries_with_callback()`
without locking the whole instance of `ConfirmationTiming`, passing just
the `BatchExecutionTiming` part.
I want to add a new metric that starts at the beginning of the
`confirm_slot_entries()` call and ends until the very end. In order to
use a `scopeguard::defer`, I need to be able to have an excursive
reference to it for the whole body of `confirm_slot_entries()`.
Plus a few minor renamings to clarify which verifications and results
variables actually store. And corrected a few messages, that
incorrectly stated PoH verification, while they were actually issued
for transaction verification failures.
* Add RewardsMessage enum
* Cache and update max_complete_rewards_slot
* Plumb max_complete_rewards_slot into JsonRpcRequestProcesseor
* Use max_complete_rewards_slot to check get_block requests
* Use max_complete_rewards_slot to limit Bigtable uploads
* Plumb max_complete_rewards_slot into RpcSubscriptions
* Use max_complete_rewards_slot to limit block subscriptions
* Nit: fix test
* Add fully-reproducible online tracer for banking
* Don't use eprintln!()...
* Update programs/sbf/Cargo.lock...
* Remove meaningless assert_eq
* Group test-only code under aptly named mod
* Remove needless overflow handling in receive_until
* Delay stat aggregation as it's possible now
* Use Cow to avoid needless heap allocs
* Properly consume metrics action as soon as hold
* Trace UnprocessedTransactionStorage::len() instead
* Loosen joining api over type safety for replaystage
* Introce hash event to override these when simulating
* Use serde_with/serde_as instead of hacky workaround
* Update another Cargo.lock...
* Add detailed comment for Packet::buffer serialize
* Rename sender_overhead_minimized_receiver_loop()
* Use type interference for TraceError
* Another minor rename
* Retire now useless ForEach to simplify code
* Use type alias as much as possible
* Properly translate and propagate tracing errors
* Clarify --enable-banking-trace with better naming
* Consider unclean (signal-based) node restarts..
* Tweak logging and cli
* Remove Bank events as it's not needed anymore
* Make tpu own banking tracer thread
* Reduce diff a bit..
* Use latest serde_with
* Finally use the published rolling-file crate
* Make test code change more consistent
* Revive dead and non-terminating test code path...
* Dispose batches early now that possible
* Split off thread handle very early at ::new()
* Tweak message for TooSmallDirByteLimitl
* Remove too much of indirection
* Remove needless pub from ::channel()
* Clarify test comments
* Avoid needless event creation if tracer is disabled
* Write tests around file rotation and spill-over
* Remove unneeded PathBuf::clone()s...
* Introduce inner struct instead of tuple...
* Remove unused enum BankStatus...
* Avoid .unwrap() for the case of disabled tracer...
Problem
The plugins need to know when all transactions for a block have been all notified to serve getBlock request correctly. As block and transaction notifications are sent asynchronously to each other it will be difficult.
Summary of Changes
Include the executed transaction count in block notification which can be used to check if all transactions have been notified.
* Plumb dumps from replay_stage to repair
When dumping a slot from replay_stage as a result of duplicate or
ancestor hashes, properly update repair subtrees to keep weighting and
forks view accurate.
* add test
* pr comments
* Add dump_node to update stake for heaviest subtrees
Additionally refactor subtrees to store children as a hashset
* Add a more complicated forks test
* chose -> choose
* remove is_dumped flag and reuse latest_invalid_ancestor instead
* Update cost model to use requested_cu instead of estimated cu #27608
* remove CostUpdate and CostModel from replay/tvu
* revive cost update service to send cost tracker stats
* CostModel is now static
* remove unused package
Co-authored-by: Tao Zhu <tao@solana.com>
We have transactions counted in replay-slot-end-to-end-stats, but that
metric is broken down to report things per thread.
So, report total_transactions for the entire slot (all threads) in
replay-slot-stats.
* Check overflow on vote tx compaction boundary
Check for overflow during the conversion between VoteStateUpdate and
CompactVoteStateUpdate.
* Try removing clippy supress
Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case
* Keypair: implement clone()
This was not implemented upstream in ed25519-dalek to force everyone to
think twice before creating another copy of a potentially sensitive
private key in memory.
See https://github.com/dalek-cryptography/ed25519-dalek/issues/76
However, there are now 9 instances of
Keypair::from_bytes(&keypair.to_bytes())
in the solana codebase and it would be preferable to have a function.
In particular since this also comes up when writing programs and can
cause users to either start messing with lifetimes or discover the
from_bytes() workaround themselves.
This patch opts to not implement the Clone trait. This avoids automatic
use in order to preserve some of the original "let developers think
twice about this" intention.
* Use Keypair::clone
* Concurrent replay slots
* Split out concurrent and single bank replay paths
* Sub function processing of replay results for readability
* Add feature switch for concurrent replay
* allow initial hash calc to occur in bg
* validator_initialized -> startup_verification_complete
* add infos for leader and vote
* rework snapshot for startup verification
* change to assert
* Make sure to root local slots even with hard fork
* Address review comments
* Cleanup a bit
* Further clean up
* Further clean up a bit
* Add comment
* Tweak hard fork reconciliation code placement
* Define shuffle to prep using same shuffle for multiple slices
* Determine transaction indexes and plumb to execute_batch
* Pair transaction_index with transaction in TransactionStatusService
* Add new ReplicaTransactionInfoVersion
* Plumb transaction_indexes through BankingStage
* Prepare BankingStage to receive transaction indexes from PohRecorder
* Determine transaction indexes in PohRecorder; add field to WorkingBank
* Add PohRecorder::record unit test
* Only pass starting_transaction_index around PohRecorder
* Add helper structs to simplify test DashMap
* Pass entry and starting-index into process_entries_with_callback together
* Add tx-index checks to test_rebatch_transactions
* Revert shuffle definition and use zip/unzip
* Only zip/unzip if randomize
* Add confirm_slot_entries test
* Review nits
* Add type alias to make sender docs more clear
* Revert "core: partial versioned transaction support for voting service"
This reverts commit eb3df4c20e.
* Manually serialize vote tx before sending to TPU
* Add simulation detection countermeasures
* Add program and test using TestValidator
* Remove incinerator deposit
* Remove incinerator
* Update Cargo.lock
* Add more features to simulation bank
* Update Cargo.lock per rebase
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
* Add feature gate for new vote instruction and plumb through replay
Add tower versions
* Add check for slot hashes history
* Update is_recent check to exclude voting on hard fork root slot
* Move tower rollback test to flaky and ignore it until #22551 lands
* Refactor: Rename leader_first_tick_height field
* Refactor: add `PohRecorder::slot_for_tick_height` helper
* Refactor: Add type for poh leader status
* metrics for generate new bank forks
* fixed
* Apply suggestions from code review
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* --fixup
* fixup!
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
* Fixup typo
* Add new feature
* Add new TransactionError
* Add framework for checking account state before and after transaction processing
* Fail transactions that leave new rent-paying accounts
* Only check rent-state of writable tx accounts
* Review comments: combine process_result success behavior; log and metrics before feature activation
* Fix tests that assume rent-exempt accounts are okay
* Remove test no longer relevant
* Remove native/sysvar special case
* Move metrics submission to report legacy->legacy rent paying transitions as well
Andrew Fitzgerald
steviez
Trent Nelson
Ashwin Sekar
Brennan
Tyera
Illia Bobyr
Ryo Onodera
behzad nouri
Kirill Fomichev
Lijun Wang
Jason Davis
Maximilian Schneider
Brooks Prumo
carllin
Justin Starry
Christian Kamm
apfitzge
Tao Zhu
Michael Vines
Jeff Biseda
Boqin Qin(秦 伯钦)
Jeff Washington (jwash)
Nicholas Clarke
sakridge
HaoranYi
anatoly yakovenko
Tyera Eulberg