Commit Graph

335 Commits

Author SHA1 Message Date
Tyera Eulberg 637e366b18
Prevent rent-paying account creation (#22292)
* Fixup typo

* Add new feature

* Add new TransactionError

* Add framework for checking account state before and after transaction processing

* Fail transactions that leave new rent-paying accounts

* Only check rent-state of writable tx accounts

* Review comments: combine process_result success behavior; log and metrics before feature activation

* Fix tests that assume rent-exempt accounts are okay

* Remove test no longer relevant

* Remove native/sysvar special case

* Move metrics submission to report legacy->legacy rent paying transitions as well
2022-01-11 11:32:25 -07:00
Jeff Biseda 8b66625c95
convert std::sync::mpsc to crossbeam_channel (#22264) 2022-01-11 02:44:46 -08:00
Justin Starry 52d12cc802
Add runtime support for address table lookups (#22223)
* Add support for address table lookups in runtime

* feedback

* feedback
2022-01-07 11:59:09 +08:00
Trent Nelson 7d32909e17 move `ExecuteTimings` from `runtime::bank` to `program_runtime::timings` 2022-01-06 03:56:46 -07:00
Justin Starry 45458e7139
Refactor: Improve type safety and readability of transaction execution (#22215)
* Refactor Bank::load_and_execute_transactions

* Refactor: improve type safety of TransactionExecutionResult

* Add enum for extra type safety in execution results

* feedback
2022-01-05 10:15:15 +08:00
behzad nouri 4b24499916 removes total-size from return value of recv_mmsg 2022-01-04 21:06:59 +00:00
behzad nouri 01a096adc8 adds bitflags to Packet.Meta
Instead of a separate bool type for each flag, all the flags can be
encoded in a type-safe bitflags encoded in a single u8:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L19-L31
2022-01-04 13:53:40 +00:00
carllin 33d0b5e011
Revert "Count compute units even when transaction errors (#22059)" (#22174)
This reverts commit eaa8c67bde.
2021-12-30 02:42:32 -05:00
Justin Starry b1d9a2e60e
Don't forward packets received from TPU forwards port (#22078)
* Don't forward packets received from TPU forwards port

* Add banking stage test
2021-12-29 19:34:31 +01:00
carllin eaa8c67bde
Count compute units even when transaction errors (#22059) 2021-12-28 17:05:11 -05:00
Tao Zhu 3d6ab96587 push live packets straight to buffer, leader only process packets from buffer 2021-12-28 15:21:24 -06:00
Yueh-Hsuan Chiang b89cd8cd1a
Avoid cloning Vec<Entry> when calling entries_to_test_shreds() (#22093) 2021-12-24 12:32:43 -08:00
Justin Starry 93c776ce19
Refactor packet deduplication and harden bench test (#22080) 2021-12-22 23:05:10 -06:00
Tao Zhu dd80a525ef
Leader QoS service metrics (#21708)
* - qos_service metrics tagged with leader thread ids to separate gossip/tpu votes and transactions;
- qos_service metrics is reported with bank slot;
- replaced timer-based reporting with signal via channel; removed async report test as qos_service now lives within a thread

* - add tpu live packets (eg, not buffered packets) states to qos metrics reporting
2021-12-22 21:39:59 +00:00
Tao Zhu 9c5d82557a skip reporting all-zero stats 2021-12-21 16:20:36 -06:00
Justin Starry 6ff0be6a82
Clean up demote program write lock feature (#21949)
* Clean up demote program write lock feature

* fix test
2021-12-16 17:27:22 -05:00
Jeff Washington (jwash) 90f41fd9b7
use cost model to limit new account creation (#21369)
* use cost model to limit new account creation

* handle every system instruction

* remove &

* simplify match

* simplify match

* add datapoint for account data size

* add postgres error handling

* handle accounts:unlock_accounts
2021-12-12 14:57:18 -06:00
Justin Starry 254ef3e7b6
Rename Packets to PacketBatch (#21794) 2021-12-11 09:44:15 -05:00
Michael Vines b8837c04ec Reformat imports to a consistent style for imports
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
2021-12-03 09:19:13 -08:00
Tao Zhu 9edfc5936d
Refactor accounts.rs with Justin's comments to improve lock accounts (#21406)
with results code path.
- fix a bug that could unlock accounts that weren't locked
- add test to the refactored function
- skip enumerating transaction accounts if qos results is an error
- add #[must_use] annotation
- avoid clone error in results
- add qos error code to unlock_accounts match statement
- remove unnecessary AbiExample
2021-11-23 21:17:55 +00:00
Lijun Wang c29838fce1
Accountsdb plugin transaction part 3: Transaction Notifier (#21374)
The TransactionNotifierInterface interface for notifying transactions.
Changes to transaction_status_service to notify the notifier of the transaction data.
Interface to query the plugin's interest in transaction data
2021-11-23 09:55:53 -08:00
Tao Zhu 0ca255220e
- Encapsulate QoS Service metrics reporting within QosServioce, so client (#21191)
code (eg banking_stage) doesn't need to worry about it.
- Remove dead cost_* stats from banking_stage, clean up call path.
2021-11-18 15:35:30 -06:00
Tao Zhu 11153e1f87
refactor cost calculation (#21062)
* - cache calculated transaction cost to allow sharing;
- atomic cost tracking op;
- only lock accounts for transactions eligible for current block;
- moved qos service and stats reporting to its own model;
- add cost_weight default to neutral (as 1), vote has zero weight;

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* Update core/src/qos_service.rs

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* Update core/src/qos_service.rs

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-11-12 01:04:53 -06:00
Justin Starry 140a5f633d
Simplify replay vote tracking by using packet metadata (#21112) 2021-11-03 09:02:48 +00:00
Justin Starry 036d7fcc81
Clean up sanitized tx creation for tests (#21006) 2021-10-27 18:09:16 +01:00
sakridge 261dd96ae3
Swap banking stage vote channels (#20987) 2021-10-26 21:20:31 +02:00
Tao Zhu c2bfce90b3
- cost_tracker is data member of a bank, it can report metrics when bank is frozen (#20802)
- removed cost_tracker_stats and histogram
- move stats reporting outside of bank freeze
2021-10-24 22:19:23 -05:00
Tao Zhu 71d0bd4605
Add counter for dropped duplicated packets, fix dropped_packets_count (#20834) 2021-10-21 02:56:48 +00:00
Tao Zhu 7496b5784b
- make cost_tracker a member of bank, remove shared instance from TPU; (#20627)
- decouple cost_model from cost_tracker; allowing one cost_model
  instance being shared within a validator;
- update cost_model api to calculate_cost(&self...)->transaction_cost
2021-10-19 14:37:33 -05:00
Tao Zhu 005d6863fd
- move cost tracker into bank, so each bank has its own cost tracker; (#20527)
- move related modules to runtime
2021-10-12 08:51:33 -05:00
Tao Zhu 0ebd8c53ee
cost model to ignore vote transactions (#20510) 2021-10-07 12:49:07 -05:00
Tao Zhu 177a375479
Tpu vote 1.7 (#20187) (#20494)
* Add separate vote processing tpu port

* Add feature to send to tpu vote port

* Add vote rejecting sigverify mode

* use packet.meta.is_simple_vote_tx in place of deserialization

* consolidate code that identifies vote tx atcommon path for cpu and gpu

* new key for feature set

* banking forward tpu vote

* add tpu vote port to dockerfile and other review changes

* Simplify thread id compare

* fix a test; updated cluster_info ABI change

Co-authored-by: Tao Zhu <tao@solana.com>

Co-authored-by: sakridge <sakridge@gmail.com>
2021-10-07 09:38:23 +00:00
Tao Zhu 03913f6661
add tx count and thread id to stats, each stat reports and resets when slot changes (#20451) 2021-10-06 00:09:19 -05:00
Tao Zhu 6ff508c643
add transaction cost histogram metrics (#20350) 2021-10-05 08:57:39 -05:00
Jeff Biseda 3854cfaa00
Use batch_send in forward_buffered_packets (#20330) 2021-09-29 20:49:43 -07:00
sakridge 013e1d9d49
Limit transaction forwarding from banking_stage (#19940) 2021-09-21 08:49:41 -07:00
carllin e6b4dd3866
Add bank to banking stage regardless of if there is a working bank (#19855) 2021-09-17 16:55:53 -07:00
sakridge dc69cc1ae4
Only allow votes when root distance gets too high (#19917) 2021-09-16 15:12:26 +02:00
Justin Starry ca3f147670
Add banking metrics for buffered and dropped packets (#19902) 2021-09-15 15:53:55 -05:00
Justin Starry 34c1a9ac85
Report consumed_buffered_packets_count stat to metrics (#19900) 2021-09-15 14:19:39 +00:00
carllin 87a7f00926
Track reset bank in PohRecorder (#19810) 2021-09-13 16:55:35 -07:00
sakridge 3a8c678f62
Remove some copying (#19691) 2021-09-08 18:32:38 +02:00
Sean Young d461a9ac10 verify_precompiles needs FeatureSet
Rather than pass in individual features, pass in the entire feature set
so that we can add the ed25519 program feature in a later commit.
2021-09-05 18:59:37 +01:00
Tyera Eulberg decec3cd8b
Demote write locks on transaction program ids (#19593)
* Add feature

* Demote write lock on program ids

* Fixup bpf tests

* Update MappedMessage::is_writable

* Comma nit

* Review comments
2021-09-04 03:05:30 +00:00
Jack May 3ec33e7d02
Fail secp256k1 if the instruction data looks incorrect (#19300) 2021-08-19 13:13:54 -07:00
Justin Starry c50b01cb60
Store versioned transactions in the ledger, disabled by default (#19139)
* Add support for versioned transactions, but disable by default

* merge conflicts

* trent's feedback

* bump Cargo.lock

* Fix transaction error encoding

* Rename legacy_transaction method

* cargo clippy

* Clean up casts, int arithmetic, and unused methods

* Check for duplicates in sanitized message conversion

* fix clippy

* fix new test

* Fix bpf conditional compilation for message module
2021-08-17 15:17:56 -07:00
Jeff Washington (jwash) e368f10973
add _for_tests to new_no_wallclock_throttle (#19086) 2021-08-05 14:50:25 -05:00
Jeff Washington (jwash) a9014ceceb
Bank::default_for_tests() (#19084) 2021-08-05 11:53:29 -05:00
sakridge 84e78316b1
Write helper for multithread update (#18808) 2021-07-29 03:16:36 +02:00
behzad nouri d2d5f36a3c
adds validator flag to allow private ip addresses (#18850) 2021-07-23 15:25:03 +00:00
Justin Starry d166b9856a
Move transaction sanitization earlier in the pipeline (#18655)
* Move transaction sanitization earlier in the pipeline

* Renamed HashedTransaction to SanitizedTransaction

* Implement deref for sanitized transaction

* bring back process_transactions test method

* Use sanitized transactions for cost model calculation
2021-07-15 22:51:27 -05:00
Trent Nelson 3a85b77bb5 hijack secp256k1 enablement feature plumbing for libsecp256k1 upgrade 2021-07-15 18:43:55 +00:00
Trent Nelson 568660b402 Revert "Remove feature switch for secp256k1 program (#18467)"
This reverts commit fd574dcb3b.
2021-07-15 18:43:55 +00:00
sakridge 7f2254225e
Move entry/poh to own crate to speed up poh bench build (#18225) 2021-07-14 14:16:29 +02:00
Tao Zhu 350baece21
Explicitly sanitize program id indexes before usage
1. check transaction has valid program_id before using it to avoid possible panic;
2. change calculate_cost function signature to return Result;
3. add CostModelError enum, update return type from Result<_, str> to Result<_, CostModelError>
2021-07-13 17:29:22 -05:00
Justin Starry fd574dcb3b
Remove feature switch for secp256k1 program (#18467)
* Remove feature switch for secp256k1 program

* fix tests
2021-07-09 10:08:03 -05:00
Tao Zhu 0e039b4094
Aggregate cost_model into cost_tracker (#18374)
* * aggregate cost_model into cost_tracker, decouple it from banking_stage to prevent accidental deadlock. * Simplified code, removed unused functions

* review fixes
2021-07-06 15:41:25 +00:00
carllin 0eca92de18
Make set roots an iterator (#18357) 2021-07-01 20:02:40 -07:00
Michael Vines b6792a3328 Add ability to change the validator identity at runtime 2021-07-01 17:50:04 -07:00
Tao Zhu 9d6f1ebef4
investigate system performance test degradation (#17919)
* Add stats and counter around cost model ops, mainly:
- calculate transaction cost
- check transaction can fit in a block
- update block cost tracker after transactions are added to block
- replay_stage to update/insert execution cost to table

* Change mutex on cost_tracker to RwLock

* removed cloning cost_tracker for local use, as the metrics show clone is very expensive.

* acquire and hold locks for block of TXs, instead of acquire and release per transaction;

* remove redundant would_fit check from cost_tracker update execution path

* refactor cost checking with less frequent lock acquiring

* avoid many Transaction_cost heap allocation when calculate cost, which
is in the hot path - executed per transaction.

* create hashmap with new_capacity to reduce runtime heap realloc.

* code review changes: categorize stats, replace explicit drop calls, concisely initiate to default

* address potential deadlock by acquiring locks one at time
2021-06-28 21:34:04 -05:00
Alexander Meißner 6514096a67 chore: cargo +nightly clippy --fix -Z unstable-options 2021-06-18 10:42:46 -07:00
Lijun Wang 071b1ee3e5
Removed pub from some functions which are actually private to improve encapsulation (#18030)
Remove the pub marker to improve encapsulation. Readability improvement only, no functional impact.
2021-06-17 10:14:21 -07:00
Tao Zhu ae27fcbcda
replay stage feed back program cost (#17731)
* replay stage feeds back realtime per-program execution cost to cost model;

* program cost execution table is initialized into empty table, no longer populated with hardcoded numbers;

* changed cost unit to microsecond, using value collected from mainnet;

* add ExecuteCostTable with fixed capacity for security concern, when its limit is reached, programs with old age AND less occurrence will be pushed out to make room for new programs.
2021-06-09 17:10:59 -05:00
Tyera Eulberg 544b3c0d17
Create solana-poh and move remaining rpc modules to solana-rpc (#17698)
* Create solana-poh crate

* Move BigTableUploadService to solana-ledger

* Add solana-rpc to workspace

* Move dependencies to solana-rpc

* Move remaining rpc modules to solana-rpc

* Single use statement solana-poh

* Single use statement solana-rpc
2021-06-04 09:23:06 -06:00
Tao Zhu b000d490ce
Cost Model to limit transactions which are not parallelizeable (#16694)
* * Add following to banking_stage:
  1. CostModel as immutable ref shared between threads, to provide estimated cost for transactions.
  2. CostTracker which is shared between threads, tracks transaction costs for each block.

* replace hard coded program ID with id() calls

* Add Account Access Cost as part of TransactionCost. Account Access cost are weighted differently between read and write, signed and non-signed.

* Establish instruction_execution_cost_table, add function to update or insert instruction cost, unit tested. It is read-only for now; it allows Replay to insert realtime instruction execution costs to the table.

* add test for cost_tracker atomically try_add operation, serves as safety guard for future changes

* check cost against local copy of cost_tracker, return transactions that would exceed limit as unprocessed transaction to be buffered; only apply bank processed transactions cost to tracker;

* bencher to new banking_stage with max cost limit to allow cost model being hit consistently during bench iterations
2021-06-01 09:16:17 -05:00
Michael Vines 9541411c15 Plumb transaction-level rewards (aka "rent debits") into the `getTransaction` RPC method 2021-05-27 03:05:05 +00:00
Tyera Eulberg 9a5330b7eb
Move gossip modules into solana-gossip crate (#17352)
* Move gossip modules to solana-gossip

* Update Protocol abi digest due to move

* Move gossip benches and hook up CI

* Remove unneeded Result entries

* Single use statements
2021-05-26 09:15:46 -06:00
Tao Zhu 0781fe1b4f
Upgrade Rust to 1.52.0 (#17096)
* Upgrade Rust to 1.52.0
update nightly_version to newly pushed docker image
fix clippy lint errors
1.52 comes with grcov 0.8.0, include this version to script

* upgrade to Rust 1.52.1

* disabling Serum from downstream projects until it is upgraded to Rust 1.52.1
2021-05-19 09:31:47 -05:00
carllin b5d30846d6
Retry latest vote if expired (#16735) 2021-04-28 11:46:16 -07:00
Michael Vines c8b474cd0b Send votes to next leader's TPU instead of our TPU 2021-04-20 00:38:21 -07:00
Michael Vines a911ae00ba clippy 2021-04-18 20:55:02 -07:00
carllin f0c150cfb9
Fix channel panic in tests (#16503)
* Fix channel panic

* Add exit signal to PohRecorder because Crossbeam doesnt drop objects inside dropped channel
2021-04-14 12:07:21 -05:00
Justin Starry 85eb37fab0
Merge pull request from GHSA-8v47-8c53-wwrc
* Track transaction check time separately from account loads

* banking packet process metrics

* Remove signature clone in status cache lookup

* Reduce allocations when converting packets to transactions

* Add blake3 hash of transaction messages in status cache

* Bug fixes

* fix tests and run fmt

* Address feedback

* fix simd tx entry verification

* Fix rebase

* Feedback

* clean up

* Add tests

* Remove feature switch and fall back to signature check

* Bump programs/bpf Cargo.lock

* clippy

* nudge benches

* Bump `BankSlotDelta` frozen ABI hash`

* Add blake3 to sdk/programs/Cargo.lock

* nudge bpf tests

* short circuit status cache checks

Co-authored-by: Trent Nelson <trent@solana.com>
2021-04-13 00:28:08 -06:00
Christian Drappi 54a04bac3d
Apple M1 compatibility (#16346)
Co-authored-by: Christian Drappi <christiandrappi@Christians-MacBook-Pro.local>
2021-04-09 17:21:01 -07:00
carllin 1219842a96
No wallclock throttle tests (#16396) 2021-04-05 19:40:16 -07:00
behzad nouri 701fc93343
patches bug in banking stage where buffered packets are never retained (#16276)
banking_stage::handle_forwarding is retaining buffered packets with
empty index, so nothing is held:
https://github.com/solana-labs/solana/blob/6f3926b64/core/src/banking_stage.rs#L520
2021-04-05 12:46:21 +00:00
behzad nouri 3f63ed9a72
removes OrderedIterator and transaction batch iteration order (#16153)
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817

However load_and_execute_transactions is iterating over
  lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.

If iteratorion_order is [1, 2, 0], and i is 0, then:
  lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
  transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).

This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.
2021-03-31 23:59:19 +00:00
Tyera Eulberg 433f1ead1c
Rpc: enable getConfirmedBlock and getConfirmedTransaction to return confirmed (not yet finalized) data (#16142)
* Add Blockstore block and tx apis that allow unrooted responses

* Add TransactionStatusMessage, and send on bank freeze; also refactor TransactionStatusSender

* Track highest slot with tx-status writes complete

* Rename and unpub fn

* Add commitment to GetConfirmed input configs

* Support confirmed blocks in getConfirmedBlock

* Support confirmed txs in getConfirmedTransaction

* Update sigs-for-addr2 comment

* Enable confirmed block in cli

* Enable confirmed transaction in cli

* Review comments

* Rename blockstore method
2021-03-26 16:47:35 -06:00
sakridge 96ccc40f0a
Set ticks_per_slot higher for banking stage tests (#16094)
Tests are timing out because the bank hit the MaxTickHeight and
will not process the transactions.
2021-03-24 14:05:43 -07:00
Jeff Washington (jwash) 57ba86c821
eliminate lock on record (#15929)
* eliminate lock on record

* use same error as MaxHeightReached

* clippy

* review feedback

* refactor should_tick code

* pr feedback
2021-03-23 09:10:04 -05:00
carllin f548a04fae
Allow unbounded wallclock processing time in tests (#15961) 2021-03-17 15:48:50 -07:00
Michael Vines 8a9b51952e Ignore flaky test_banking_stage_entries_only and test_banking_stage_entryfication 2021-03-17 11:28:56 -07:00
carllin c1ba265dd9
Wallclock BankingStage Throttle (#15731) 2021-03-15 17:11:15 -07:00
sakridge d09112fa6d
PoH batch size calibration (#15717) 2021-03-05 16:01:21 -08:00
Jeff Washington (jwash) be35c1c1b7
add execute detail timings (#15638) 2021-03-03 17:07:45 -06:00
carllin aacb28c453
Only report metrics every second (#15652) 2021-03-03 10:58:47 -08:00
sakridge 830be855dc
Forward and hold packets (#15634) 2021-03-03 10:23:05 -08:00
behzad nouri 0bd0084b0d
adds more metrics for tx counts and batch sizes (#15642) 2021-03-03 01:28:15 +00:00
Michael Vines 5df36aec7d Pacify clippy 2021-02-19 20:08:41 -08:00
Tyera Eulberg 170cb792eb
Return blockstore error if previous_blockhash cannot be determined (#15382)
* Return blockstore error if previous_blockhash cannot be determined

* Add require_previous_blockshash flag
2021-02-18 01:04:52 +00:00
carllin 629dcd0f39
Cleanup buffered packets (#15210) 2021-02-12 03:27:37 -08:00
Tyera Eulberg cbb8b79a60
Add validator flag to opt in to cpi and logs storage (#14922)
* Add validator flag to opt in to cpi and logs storage

* Default TestValidator to opt-in; allow using in multinode-demo

* No clone

Co-authored-by: Carl Lin <carl@solana.com>
2021-02-01 14:00:51 -07:00
Ryo Onodera d6873b82ab
Remove potentially too costly Packets::default() (#14821)
* Remove potentially too costly Packets::default()

* Fix test...

* Restore Packets::default()

* Restore Packets::default() more
2021-01-29 09:32:38 +09:00
sakridge 907f518f6d
Add load/execute/store timings (#14561) 2021-01-14 14:14:16 -08:00
Trent Nelson fe667db910 validator: Add experimental flag to select PoH pinned core 2020-12-29 19:15:44 -07:00
sakridge 2074e407cd
Add poh speed check and tick speed calibration (#14292) 2020-12-29 09:35:57 -08:00
sakridge da7d1e2302
Improved Transaction Forwarding (#13944)
* Forwarding

* Dedupe leaders

* Use consistent commitment for last_valid_slot in rpc send_transaction

* Plumb rpc send_transaction options into solana-validator

* Extend num slots banking-stage holds forwarded txs

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-12-17 15:37:22 -07:00
sakridge 5294f70189
Remove some non-warnings (#14115) 2020-12-14 16:26:17 -08:00
Michael Vines 7143aaa89b Clippy 2020-12-14 08:03:29 -08:00
Josh 13db3eca9f
SPL token balance in transaction metadata (#13673)
* feat: store pre / post token balances

* move helper functions into separate include

* move token balance functionality to transaction-status crate

* fix blockstore processor test

* fix bigtable legacy test

* add caching to decimals
2020-12-10 19:25:07 -08:00