* Move transaction sanitization earlier in the pipeline
* Renamed HashedTransaction to SanitizedTransaction
* Implement deref for sanitized transaction
* bring back process_transactions test method
* Use sanitized transactions for cost model calculation
1. check transaction has valid program_id before using it to avoid possible panic;
2. change calculate_cost function signature to return Result;
3. add CostModelError enum, update return type from Result<_, str> to Result<_, CostModelError>
* Add stats and counter around cost model ops, mainly:
- calculate transaction cost
- check transaction can fit in a block
- update block cost tracker after transactions are added to block
- replay_stage to update/insert execution cost to table
* Change mutex on cost_tracker to RwLock
* removed cloning cost_tracker for local use, as the metrics show clone is very expensive.
* acquire and hold locks for block of TXs, instead of acquire and release per transaction;
* remove redundant would_fit check from cost_tracker update execution path
* refactor cost checking with less frequent lock acquiring
* avoid many Transaction_cost heap allocation when calculate cost, which
is in the hot path - executed per transaction.
* create hashmap with new_capacity to reduce runtime heap realloc.
* code review changes: categorize stats, replace explicit drop calls, concisely initiate to default
* address potential deadlock by acquiring locks one at time
* replay stage feeds back realtime per-program execution cost to cost model;
* program cost execution table is initialized into empty table, no longer populated with hardcoded numbers;
* changed cost unit to microsecond, using value collected from mainnet;
* add ExecuteCostTable with fixed capacity for security concern, when its limit is reached, programs with old age AND less occurrence will be pushed out to make room for new programs.
* Create solana-poh crate
* Move BigTableUploadService to solana-ledger
* Add solana-rpc to workspace
* Move dependencies to solana-rpc
* Move remaining rpc modules to solana-rpc
* Single use statement solana-poh
* Single use statement solana-rpc
* * Add following to banking_stage:
1. CostModel as immutable ref shared between threads, to provide estimated cost for transactions.
2. CostTracker which is shared between threads, tracks transaction costs for each block.
* replace hard coded program ID with id() calls
* Add Account Access Cost as part of TransactionCost. Account Access cost are weighted differently between read and write, signed and non-signed.
* Establish instruction_execution_cost_table, add function to update or insert instruction cost, unit tested. It is read-only for now; it allows Replay to insert realtime instruction execution costs to the table.
* add test for cost_tracker atomically try_add operation, serves as safety guard for future changes
* check cost against local copy of cost_tracker, return transactions that would exceed limit as unprocessed transaction to be buffered; only apply bank processed transactions cost to tracker;
* bencher to new banking_stage with max cost limit to allow cost model being hit consistently during bench iterations
* Move gossip modules to solana-gossip
* Update Protocol abi digest due to move
* Move gossip benches and hook up CI
* Remove unneeded Result entries
* Single use statements
* Upgrade Rust to 1.52.0
update nightly_version to newly pushed docker image
fix clippy lint errors
1.52 comes with grcov 0.8.0, include this version to script
* upgrade to Rust 1.52.1
* disabling Serum from downstream projects until it is upgraded to Rust 1.52.1
* Track transaction check time separately from account loads
* banking packet process metrics
* Remove signature clone in status cache lookup
* Reduce allocations when converting packets to transactions
* Add blake3 hash of transaction messages in status cache
* Bug fixes
* fix tests and run fmt
* Address feedback
* fix simd tx entry verification
* Fix rebase
* Feedback
* clean up
* Add tests
* Remove feature switch and fall back to signature check
* Bump programs/bpf Cargo.lock
* clippy
* nudge benches
* Bump `BankSlotDelta` frozen ABI hash`
* Add blake3 to sdk/programs/Cargo.lock
* nudge bpf tests
* short circuit status cache checks
Co-authored-by: Trent Nelson <trent@solana.com>
* Add validator flag to opt in to cpi and logs storage
* Default TestValidator to opt-in; allow using in multinode-demo
* No clone
Co-authored-by: Carl Lin <carl@solana.com>
* feat: store pre / post token balances
* move helper functions into separate include
* move token balance functionality to transaction-status crate
* fix blockstore processor test
* fix bigtable legacy test
* add caching to decimals
* introduce store program logs in blockstore / bigtable
* fix test, transaction logs created for successful transactions
* fix test for legacy bincode implementation around log_messages
* only api nodes should record logs
* truncate transaction logs to 100KB
* refactor log truncate for improved coverage
* Gossip benchmark
* Rayon tweaking
* push pulls
* fanout to max nodes
* fixup! fanout to max nodes
* fixup! fixup! fanout to max nodes
* update
* multi vote test
* fixup prune
* fast propagation
* fixups
* compute up to 95%
* test for specific tx
* stats
* stats
* fixed tests
* rename
* track a lagging view of which nodes have the local node in their active set in the local received_cache
* test fixups
* dups are old now
* dont prune your own origin
* send vote to tpu
* tests
* fixed tests
* fixed test
* update
* ignore scale
* lint
* fixup
* fixup
* fixup
* cleanup
Co-authored-by: Stephen Akridge <sakridge@gmail.com>
* Separate client types into own crate, so ledger does not need it
Removes about 50 crates of dependency from ledger
* Drop Rpc name from transaction-status types
* Rework transaction processing result forwarding
Durable nonce prereq
* Add Durable Nonce program API
* Add runtime changes for Durable Nonce program
* Register Durable Nonce program
* Concise comments and bad math
* Fix c/p error
* Add rent sysvar to withdraw ix
* Remove rent exempt required balance from Meta struct
* Use the helper
* Pass blocktree into execute_batch, if persist_transaction_status
* Add validator arg to enable persistent transaction status store
* Pass blocktree into banking_stage, if persist_transaction_status
* Add validator params to bash scripts
* Expose actual transaction statuses outside Bank; add tests
* Fix benches
* Offload transaction status writes to a separate thread
* Enable persistent transaction status along with rpc service
* nudge
* Review comments
* credit_only_credits_forwarding
* whack transfer_now()
* fixup
* bench should retry the airdrop TX
* fixup
* try to make bench-exchange a bit more robust, informative
* Stabilize some banking stage tests
Fixes#5660
* Fix CI...
* clean up
* Fix ci
* Address review nits
* Use bank.max_tick_height due to off-by-one for no PohRecord's clearing bank
* Fix CI...
* Use bank.max_tick_height() instead for clarity
Summary of Changes:
This change adds functionality to randomize tx execution for every entry. It does this by implementing OrderedIterator that iterates tx slice as per the order specified. The order is generated randomly for every entry.
* Rate limit transaction counters
* @sakridge feedback
* Set default high metrics rate for multinode demo
* Fix tests
* Swap defaults and fix env var tests
* Only set metrics rate if not already set
* Ensure signable data is not out of range
* Add a broadcast stage that puts bad sizes in blobs
* Resign blob after modifyign size
* Remove assertions that fail when size != meta.size
* Add committable transactions that cause errors like InstructionErrors back to retryable list on MaxHeightReached
* Remove unnecessary logic
* Add comments/renaming for clarity
* Implement CreditOnlyLocks
* Update credit-only atomic on account load
* Update credit-only atomic after bank.freeze_lock; store credits if all credit-only lock references are dropped
* Commit credit-only credits on bank freeze
* Update core to CreditAccountLocks
* Impl credit-only in System Transfer
* Rework CreditAccountLocks, test, and fix bugs
* Review comments: Pass CreditAccountLocks by reference; Tighten up insert block
* Only store credits on completed slot
* Check balance in bench_exchange funding to ensure commit_credits has completed
* Add is_debitable info to KeyedAccount meta to pass into programs
* Reinstate CreditOnlyLocks check on lock_account
* Rework CreditAccountLocks to remove strong_count usage
* Add multi-threaded credit-only locks test
* Improve RwLocks usage
* Review comments: panic if bad things happen; tighter code
* Assert lock_accounts race does not happen
* Revert panic if bad things happen; not a bad thing
* Page-pin packet memory for cuda
Bring back recyclers and pin offset buffers
* Add packet recycler to streamer
* Add set_pinnable to sigverify vecs to pin them
* Add packets reset test
* Add test for recycler and reduce the gc lock critical section
* Add comments/tests to cuda_runtime
* Add recycler to recv_blobs path.
* Add trace/names for debug and PacketsRecycler to bench-streamer
* Predict realloc and unpin beforehand.
* Add helper to reserve and pin
* Cap buffered packets length
* Call cuda wrapper functions
* Use Rc to prevent clone of Packets
* Fix min => max in banking_stage threads.
Coalesce packet buffers better since a larger batch will
be faster through banking and sigverify.
Deconstruct batches into banking_stage from sigverify since
sigverify likes to accumulate batches but then a single banking_stage
thread will be stuck with a large batch. Maximize parallelism by
creating more chunks of work for banking_stage.
* Remove unused PohServiceConfig::Step
* Clarify variable name
* Poh::hash() now takes an iteration counter
* man -> max
* Inline functions with single call site
* Move PohServiceConfig into GenesisBlock
* Add plumbing to enable real PoH on testnets
* Batch hashes to improve PoH hash rate
* Ensure a constant hashes_per_tick
* Remove PohEntry mixin field
* Poh/PohEntry no longer maintains tick_height
* Ensure a constant hashes_per_tick
* ci/localnet-sanity.sh: Use real PoH
* Rework Poh/PohService to keep PohRecorder unlocked as much as possible while hashing
* Introduce record locks on txs that will be recorded
* Add tests for LockedAccountsResults
* Fix broken bench
* Exit process_entries on detecting conflicting locks within same entry
* * rename Entry::serialized_size() to Entry::to_blob_size() to better
reduce confusion with bincode, et al. and to better reflect its
real meaning
* fix implementation of to_blob_size() to actually return what happens
when we do entries.to_blobs() (i.e. we serialize Vec<Entry>, not Entry)
* update tests to be more rigorous
* clippy
- packets are buffered on leader rotation, when the next leader is
unknown
- shortening the wait allows the banking stage to poll for next
leader more frequently
Use transaction count instead of batch count,
and set the recv_start from when we finished processing
the previous batch to get a more accurate number.