blockworks-foundation
/
solana
mirror of https://github.com/blockworks-foundation/solana.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
19,367 Commits 168 Branches 497 Tags 539 MiB
Commit Graph

3100 Commits

Author SHA1 Message Date
Jeff Washington (jwash) 5b916961b5
HashCalc uses self.accounts_cache (#23890) 2022-03-24 10:34:28 -05:00
Jeff Washington (jwash) b22165ad69
hash calc uses self.filler_account_suffix (#23887) 2022-03-24 09:58:06 -05:00
Jeff Washington (jwash) 9022931689
calc hash uses self.num_hash_scan_passes (#23883) 2022-03-24 09:44:42 -05:00
Jeff Washington (jwash) db5d68f01f
HashCalc uses self.accounts_hash_cache_path (#23882) 2022-03-24 09:31:55 -05:00
Jeff Washington (jwash) 3e22d4b286
calc hash uses self.thread_pool_clean (#23881) 2022-03-23 20:52:38 -05:00
Jeff Washington (jwash) 9e61fe7583
add AccountsHashConfig to manage parameters (#23850) 2022-03-23 13:44:23 -05:00
HaoranYi db49b826f0
seperate blockstore metrics from window service metrics (#23871) 2022-03-23 13:38:17 -05:00
HaoranYi 7ff8ed869c
typos (#23870) 2022-03-23 13:36:55 -05:00
Jeff Washington (jwash) b1280b670a
calculate_accounts_hash_without_index takes &self (#23846) 
* calculate_accounts_hash_without_index takes &self

* Update runtime/src/snapshot_package.rs

Co-authored-by: Brooks Prumo <brooks@prumo.org>

Co-authored-by: Brooks Prumo <brooks@prumo.org>
 2022-03-23 11:57:32 -05:00
Justin Starry 92462ae031
Manually serialize and use `send_wire_transaction` for votes (#23826) 
* Revert "core: partial versioned transaction support for voting service"

This reverts commit eb3df4c20e.

* Manually serialize vote tx before sending to TPU
 2022-03-23 09:47:55 +08:00
Jon Cinque 7af48465fa
transaction-status: Add return data to meta (#23688) 
* transaction-status: Add return data to meta

* Add return data to simulation results

* Use pretty-hex for printing return data

* Update arg name, make TransactionRecord struct

* Rename TransactionRecord -> ExecutionRecord
 2022-03-22 23:17:05 +01:00
Yueh-Hsuan Chiang ae75b1a25f
(LedgerStore) Add compression type (#23578) 
This PR adds `--rocksdb-ledger-compression` as a hidden argument to the validator
for specifying the compression algorithm for TransactionStatus.  Available compression
algorithms include `lz4`, `snappy`, `zlib`. The default value is `none`.

Experimental results show that with lz4 compression, we can achieve ~37% size-reduction
on the TransactionStatus column family, or ~8% size-reduction of the ledger store size.
 2022-03-22 02:27:09 -07:00
Trent Nelson eb3df4c20e core: partial versioned transaction support for voting service 2022-03-21 22:59:05 -06:00
HaoranYi 45a7c6edfb
Fix typos and a small refactor (#23805) 
* fix typo

* remove packet_has_more_unprocessed_transactions function
 2022-03-21 18:35:31 -05:00
Will Hickey c4ecfa5716
Bump version to v1.11 (#23807) 
* Revert crossbeam_epoch to stable. 0.9.8 only works with nightly
* Remove unneeded unit expression
 2022-03-21 17:40:50 -05:00
Pankaj Garg 5d03b188c8
Use QUIC client in voting service (#23713) 
* Use QUIC client in voting service

* guard quic-client usage with a flag

* add measure to time the quic client

* move time measure outside if block

* remove quic vs UDP flag from voting service
 2022-03-21 09:10:16 -07:00
Tao Zhu 71ea05c176 replace nested for_each with flat_map 2022-03-18 16:37:41 -05:00
Tao Zhu 1c369fb55f Scan entire UnprocessedPacketBatches buffer to produce stake and locator of each packet 2022-03-18 16:37:41 -05:00
Yueh-Hsuan Chiang f999eef452
(LedgerStore) Rename BlockstoreAdvancedOptions to LedgerColumnOptions (#23764) 
This PR renames BlockstoreAdvancedOptions to LedgerColumnOptions, as we will
pass-down this struct to LedgerColumn to allow it to perform metric reporting.
 2022-03-18 11:13:35 -07:00
Tao Zhu 56428be629 Not exposing inner cost_table to encapsulating implementation details, 
making future change easier.
 2022-03-18 12:58:43 -05:00
Tao Zhu 0ed23899e7 directly use compute_budget MAX_UNITS and DEFAULT_UNITS 2022-03-18 08:53:11 -05:00
Tao Zhu a4cacf3389 add deterministic default cost 2022-03-18 08:53:11 -05:00
Tao Zhu c478fe2047 add timing metrics, some renaming 2022-03-17 19:31:28 -05:00
Tao Zhu fd515097d8 leader qos part 2: add stage to find sender stake, set to packet meta 2022-03-17 19:31:28 -05:00
Stephen Akridge 976b138e76 Add tx weighting stage 2022-03-17 19:31:28 -05:00
Will Hickey 2f58c9e501
Bump version to 1.10.4 (#23743) 2022-03-17 14:02:13 -05:00
Michael Vines 3773b753d1 Configure shrink paths during blockstore load 2022-03-15 23:08:07 -07:00
Michael Vines ab373bb1a9 Refactor new_banks_from_ledger() into load and process steps 2022-03-15 23:08:07 -07:00
Michael Vines 2da4e3eb6c Add --no-os-memory-stats-reporting 2022-03-15 17:07:40 -07:00
Michael Vines dbc62f2e28 Use consistent variable naming for DropBankService 2022-03-15 17:07:13 -07:00
Michael Vines d44f3d7216 Remove unhelpful log message 2022-03-15 17:07:13 -07:00
Tao Zhu 2d3501dff9 make upsert infallible op 2022-03-15 17:05:41 -05:00
dependabot[bot] 62b26f012e
chore: bump reqwest from 0.11.9 to 0.11.10 (#23671) 
* chore: bump reqwest from 0.11.9 to 0.11.10

Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.9 to 0.11.10.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.9...v0.11.10)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-03-15 12:54:23 -06:00
Tao Zhu 61cead9b9b Remove injection of exit signal into cost_update_service 2022-03-15 09:58:56 -05:00
Tao Zhu eb73dacd58 harden banking tests 2022-03-15 09:58:08 -05:00
Justin Starry 8c8f9694e0
Refactor: Sanitized transaction creation (#23558) 
* Refactor: SanitizedTransaction::try_create optionally computes hash

* Refactor: Add SimpleAddressLoader
 2022-03-15 12:02:22 +08:00
Tyera Eulberg 102dd68a03
Rename AccountsDb plugins to Geyser plugins (#23604) 2022-03-14 19:18:46 -06:00
Michael Vines 17cc095d28 Slot warping doesn't need to be in new_banks_from_ledger 2022-03-14 15:29:58 -07:00
Michael Vines 2e7ee0f177 Tower loading doesn't need to be in new_banks_from_ledger 2022-03-14 15:29:58 -07:00
Michael Vines 390dc24608 Create leader schedule before processing blockstore 2022-03-14 15:29:58 -07:00
Michael Vines 543d5d4a5d Reduce new_banks_from_ledger arguments 2022-03-14 15:29:58 -07:00
Michael Vines 115f376465 Factor out bank_forks_utils::load_bank_forks() 2022-03-14 15:29:58 -07:00
Michael Vines c2ce152be8 Inline do_process_blockstore_from_root 2022-03-14 15:29:58 -07:00
Tao Zhu 5ea6a1e500 code review 2022-03-14 13:14:27 -05:00
Tao Zhu 8590911b0a Replace type alias with newtype for UnprocesedPacketBatches 2022-03-14 13:14:27 -05:00
Will Hickey 63bf0f66af
Bump version to 1.10.3 (#23648) 2022-03-14 11:18:45 -05:00
Brooks Prumo 7758c32035
Banking Stage drops transactions that'll exceed the total account data size limit (#23537) 2022-03-13 15:58:57 +00:00
Yueh-Hsuan Chiang 1e20bd8f9a
(LedgerStore) Include storage type as a tag in RocksDB metric reporting (#23523) 
#### Summary of Changes
This PR further enables group by operation on storage type in blockstore_rocksdb_cfs metrics.
Such group-by allows us to further compare the performance metrics between rocks-level and
rocks-fifo.

To make things extensible, this PR introduces BlockstoreAdvancedOptions and move shred_storage_type. 
All fields in BlockstoreAdvancedOptions will support group-by operation in blockstore_rocksdb_cfs.

Dependency: #23580
 2022-03-11 15:17:34 -08:00
dependabot[bot] b1da7cff66
chore: bump base64 from 0.12.3 to 0.13.0 (#23616) 
* chore: bump base64 from 0.12.3 to 0.13.0

Bumps [base64](https://github.com/marshallpierce/rust-base64) from 0.12.3 to 0.13.0.
- [Release notes](https://github.com/marshallpierce/rust-base64/releases)
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/marshallpierce/rust-base64/compare/v0.12.3...v0.13.0)

---
updated-dependencies:
- dependency-name: base64
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update Cargo.lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2022-03-11 16:01:15 -07:00
Brooks Prumo d20dd21600
Sort tables in Cargo.toml files (#23602) 2022-03-11 11:05:57 -06:00
sakridge 9b591286d7
Revert "chore: bump dashmap from 4.0.2 to 5.1.0 (#23372)" (#23592) 
This reverts commit 3a0271c113.
 2022-03-11 09:12:27 +01:00
Will Hickey b444836a97
Bump version to 1.10.2 (#23597) 2022-03-10 16:41:06 -06:00
Tao Zhu 35d1235ed0
- move `unprocessed_packet_batches` from `BankingStage` to its own (#23508) 
module
- deserialize packets during receving and buffering
 2022-03-10 18:47:46 +00:00
carllin 588414a776
Report even if slot begins and ends in process_buffered_packets() (#23549) 2022-03-09 23:42:35 -05:00
Tao Zhu f68c5a274d remove persist_cost_table code 2022-03-09 21:05:47 -07:00
Tao Zhu 9f71958d7d Patch validator from loading persisted program costs 2022-03-09 21:05:47 -07:00
Brooks Prumo 9bbccbe27c
Use `AsRef<Path>` instead of `PathBuf` for parameters (#23560) 2022-03-09 16:08:33 -06:00
HaoranYi a1c45d5acb
typo (#23564) 2022-03-09 15:11:21 -06:00
sakridge 7a9884c831
Quic limit connections (#23283) 
* quic server limit connections

* bump per_ip

* Review comments

* Make the connections per port
 2022-03-09 10:52:31 +01:00
dependabot[bot] 5c722519cf chore: bump etcd-client from 0.8.3 to 0.8.4 
Bumps [etcd-client](https://github.com/etcdv3/etcd-client) from 0.8.3 to 0.8.4.
- [Release notes](https://github.com/etcdv3/etcd-client/releases)
- [Commits](https://github.com/etcdv3/etcd-client/compare/0.8.3...v0.8.4)

---
updated-dependencies:
- dependency-name: etcd-client
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-08 17:33:24 -08:00
Carl Lin 5a0cd05866 Revert "- estimate a program cost as 2 standard deviation above mean" 
This reverts commit a25ac1c988.
 2022-03-08 17:18:44 -08:00
Carl Lin 9acbfa5eb1 Revert "use EMA in place of Welford" 
This reverts commit 6587dbfa47.
 2022-03-08 17:18:44 -08:00
Carl Lin c878c9e2cb Revert "1. Persist to blockstore less frequently;" 
This reverts commit 7aa1fb4e24.
 2022-03-08 17:18:44 -08:00
Carl Lin 0a17edcc1f Revert "fix tests after merge" 
This reverts commit ba2d83f580.
 2022-03-08 17:18:44 -08:00
Michael Vines b719d6a2ad `solana-validator set-identity` no longer writes a tower file unnecessarily 2022-03-08 15:34:23 -08:00
Justin Starry 3114c199bd
Add RPC support for versioned transactions (#22530) 
* Add RPC support for versioned transactions

* fix doc tests

* Add rpc test for versioned txs

* Switch to preflight bank
 2022-03-08 15:20:34 +08:00
dependabot[bot] 38d8bbb19c
chore: bump sysctl from 0.4.3 to 0.4.4 (#23505) 
Bumps [sysctl](https://github.com/johalun/sysctl-rs) from 0.4.3 to 0.4.4.
- [Release notes](https://github.com/johalun/sysctl-rs/releases)
- [Changelog](https://github.com/johalun/sysctl-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/johalun/sysctl-rs/commits)

---
updated-dependencies:
- dependency-name: sysctl
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-07 14:24:19 -07:00
HaoranYi 181fffb916
rename status filename to be consistent (#23501) 2022-03-07 17:34:35 +00:00
dependabot[bot] 3a0271c113
chore: bump dashmap from 4.0.2 to 5.1.0 (#23372) 
* chore: bump dashmap from 4.0.2 to 5.1.0

Bumps [dashmap](https://github.com/xacrimon/dashmap) from 4.0.2 to 5.1.0.
- [Release notes](https://github.com/xacrimon/dashmap/releases)
- [Commits](https://github.com/xacrimon/dashmap/commits/v5.1.0)

---
updated-dependencies:
- dependency-name: dashmap
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-03-07 00:00:26 -05:00
Yueh-Hsuan Chiang b8b7163b66
(Ledger Store) Report RocksDB Column Family Metrics (#22503) 
This PR enables blockstore to periodically report RocksDB column family properties.
The reported properties are under blockstore_rocksdb_cfs, and the properties also
support group by operation on cf_name.
 2022-03-05 16:13:03 -08:00
Yueh-Hsuan Chiang 62d2a4cd88
Make ShredStorageType::RocksLevel public (#23272) 
#### Summary of Changes
This PR adds two hidden arguments to the validator that allow users to use RocksDB's FIFO compaction for storing shreds.

        --shred-storage <SHRED_STORAGE>
            EXPERIMENTAL: Controls how RocksDB compacts shreds.  *WARNING*: You will lose your ledger data
            when you switch between options. Possible values are: 'level': stores shreds using RocksDB's default (level)
            compaction. 'fifo': stores shreds under RocksDB's FIFO compaction. This option is more efficient on
            disk-write-bytes of the ledger store. [default: level]  [possible values: level, fifo]

        --shred-storage-size <SHRED_STORAGE_SIZE_BYTES>
            The shred storage size in bytes. The suggested value is 50% of your ledger storage size in bytes. [default:
            268435456000]
 2022-03-03 12:43:58 -08:00
dependabot[bot] f2fa49a771
chore: bump lru from 0.7.2 to 0.7.3 (#23462) 
Bumps [lru](https://github.com/jeromefroe/lru-rs) from 0.7.2 to 0.7.3.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases)
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jeromefroe/lru-rs/compare/0.7.2...0.7.3)

---
updated-dependencies:
- dependency-name: lru
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-03 12:01:49 -07:00
dependabot[bot] 61d7bdd66f
chore: bump serde_json from 1.0.78 to 1.0.79 (#23461) 
* chore: bump serde_json from 1.0.78 to 1.0.79

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.78 to 1.0.79.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.78...v1.0.79)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2022-03-02 23:38:06 -07:00
dependabot[bot] fe7604589d
chore: bump retain_mut from 0.1.5 to 0.1.7 (#23450) 
Bumps [retain_mut](https://github.com/upsuper/retain_mut) from 0.1.5 to 0.1.7.
- [Release notes](https://github.com/upsuper/retain_mut/releases)
- [Commits](https://github.com/upsuper/retain_mut/commits)

---
updated-dependencies:
- dependency-name: retain_mut
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-02 23:15:33 +00:00
Will Hickey 1a99251498
Bump version to 1.10.1 (#23453) 2022-03-02 13:47:01 -06:00
Jeff Washington (jwash) 26aa18b3f3
fmt (#23448) 2022-03-02 11:54:58 -06:00
HaoranYi 41f78b9925
small optimization. use shift for pow of 2. (#22975) 2022-03-02 09:11:12 -06:00
HaoranYi 4f0070a5c6
unittest for bind two consecutive ports (#23008) 
* minor fix of comments in fork-selection tests

* fix doc link

* add unittest for bind_two_consecutive_in_range
 2022-03-02 09:10:29 -06:00
HaoranYi 8de88d0a55
Refactor packet_threshold adjustment code into its own struct (#23216) 
* refactor packet_threshold adjustment code into own struct and add unittest for it

* fix a typo in error message

* code review feedbacks

* another code review feedback

* Update core/src/ancestor_hashes_service.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* share packet threshold with repair service (credit to carl)

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
 2022-03-02 09:09:06 -06:00
HaoranYi 86e2f728c3
Fix a batch limits bug in banking (#23327) 
* add thread index in thread name for debugging

* fix batch_limit

* use NUM_VOTE_THREAD instead of hardcoded number (credit to carllin)
 2022-03-02 09:08:08 -06:00
Tyera Eulberg d3ebe8d8f5
Remove unneeded jsonrpc dependencies/features; update do-audit (#23436) 
* Update generic-array note

* Remove unneeded jsonrpc deps

* Remove unneeded jsonrpc features

* Rewrite slot-update test without websocket crate

* Rewrite rpc-subscription test without websocket crate, and remove jsonrpc deps

* Update expected balance to accommodate rent-exempt minimum transfer amount

* Remove obsolete audit ignores
 2022-03-02 01:42:01 -07:00
Jeff Biseda c69e3b73ff
bench get_retransmit_peers (#23292) 2022-03-01 19:10:29 -08:00
dependabot[bot] ce4d579499
chore: bump serial_test from 0.5.1 to 0.6.0 (#23414) 
Bumps [serial_test](https://github.com/palfrey/serial_test) from 0.5.1 to 0.6.0.
- [Release notes](https://github.com/palfrey/serial_test/releases)
- [Commits](https://github.com/palfrey/serial_test/compare/v0.5.1...v0.6.0)

---
updated-dependencies:
- dependency-name: serial_test
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-01 12:13:40 -07:00
Brooks Prumo 533eca3b4c
Simplify replay_blockstore_into_bank() (#23282) 2022-02-25 06:57:04 -06:00
Trent Nelson d4292774c5 checks 2022-02-25 08:05:28 +00:00
dependabot[bot] 98f059e89c
chore: bump reqwest from 0.11.6 to 0.11.9 (#23337) 
* chore: bump reqwest from 0.11.6 to 0.11.9

Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.6 to 0.11.9.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.6...v0.11.9)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-02-24 22:40:01 -07:00
Justin Starry d0e85c293f
Fix rustfmt check (#23296) 2022-02-23 16:38:53 +08:00
Gavin Chan 20d031e2b8
Refactor ExecuteTimings w/ enum-indexed array (#23085) 2022-02-22 14:46:56 -08:00
Tyera Eulberg 7e08ae1d0c
Revert "Add simulation detection countermeasure (#22880)" (#23261) 
This reverts commit c42b80f099.
 2022-02-21 21:15:37 +00:00
Yueh-Hsuan Chiang 8c872e9ce0
(LedgerStore/FIFO) Refactor FIFO options and sanity check. (#23131) 2022-02-19 00:58:38 -08:00
buffalu 70ebab2c82
Add rustfmt.toml and `cargo fmt` (#23238) 
* fmt

* formatted

Co-authored-by: Lucas B <buffalu@jito.network>
 2022-02-19 13:32:29 +08:00
Yueh-Hsuan Chiang 1add82aa9e
(Ledger Store Benchmark) Add flags for checking shred storage size. (#22451) 2022-02-18 19:35:28 -08:00
carllin 619335df1a
Add execute timings (#23097) 2022-02-17 01:14:32 -05:00
anatoly yakovenko 83d31c9e65
shrink batches when over 80% of the space is wasted (#23066) 
* shrink batches when over 80% of the space is wasted
 2022-02-16 08:18:17 -08:00
Jeff Biseda 115d71536b
forward_buffered_packets return packet count in error path (#23167) 2022-02-16 07:46:32 -08:00
Michael Vines a6d736572c `solana-validator set-identity` now supports the `--require-tower` flag 2022-02-15 19:45:00 -08:00
Tao Zhu 03bf66a51b flag end-of-slot when poh bank is gone 2022-02-15 15:01:27 -06:00
Ashwin Sekar ab92578b02
Fix the flaky test test_restart_tower_rollback (#23129) 
* Add flag to disable voting until a slot to avoid duplicate voting

* Fix the tower rollback test and remove it from flaky.
 2022-02-15 13:19:34 -07:00
Michael Vines c42b80f099
Add simulation detection countermeasure (#22880) 
* Add simulation detection countermeasures

* Add program and test using TestValidator

* Remove incinerator deposit

* Remove incinerator

* Update Cargo.lock

* Add more features to simulation bank

* Update Cargo.lock per rebase

Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
 2022-02-15 13:09:59 +01:00
Yueh-Hsuan Chiang 824446710b
(Ledger Store Benchmark) Enable FIFO compmaction in the benchmark. (#22162) 2022-02-12 00:43:25 -08:00
Lijun Wang c04438be4b
Retaining transaction logs when transaction plugin is loaded. (#22874) 
Transaction logs are not being saved to the database through the plugin interface.

Summary of Changes

Retain the transaction logs when transaction notification plugin is loaded.

Fixes #
lijunwangs/solana-accountsdb-plugin-postgres#6
 2022-02-11 20:29:07 -08:00
carllin 2f9e30a1f7
Introduce slot-specific packet metrics (#22906) 2022-02-11 03:07:45 -05:00
dependabot[bot] 59e64d5e99
chore: bump raptorq from 1.6.4 to 1.6.5 (#23042) 
Bumps [raptorq](https://github.com/cberner/raptorq) from 1.6.4 to 1.6.5.
- [Release notes](https://github.com/cberner/raptorq/releases)
- [Commits](https://github.com/cberner/raptorq/compare/v1.6.4...v1.6.5)

---
updated-dependencies:
- dependency-name: raptorq
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-10 19:07:07 -07:00
Justin Starry d5dec989b9
Enforce tx metadata upload with static types (#23028) 2022-02-10 13:28:18 +08:00
Yueh-Hsuan Chiang 1b287f1b59
(Ledger Cleanup) Add code comments for ledger_cleanup. (#22807) 2022-02-08 22:48:56 -08:00
Tao Zhu ba2d83f580 fix tests after merge 2022-02-08 16:18:23 -06:00
Tao Zhu 7aa1fb4e24 1. Persist to blockstore less frequently; 
2. reduce alpha for EMA to 1 percent to have roughly 200 data points for estimatio
 2022-02-08 16:18:23 -06:00
Tao Zhu 6587dbfa47 use EMA in place of Welford 2022-02-08 16:18:23 -06:00
Tao Zhu a25ac1c988 - estimate a program cost as 2 standard deviation above mean 
- replaced get_average / get_mode with get_default to assign max units to unknown program
 2022-02-08 16:18:23 -06:00
Tao Zhu e52e48076e
bench should update leader schedule cache (#22991) 2022-02-08 02:28:28 +00:00
Ashwin Sekar 5acf0f6331
Add feature gate for new vote instruction and plumb through replay (#21683) 
* Add feature gate for new vote instruction and plumb through replay

Add tower versions

* Add check for slot hashes history

* Update is_recent check to exclude voting on hard fork root slot

* Move tower rollback test to flaky and ignore it until #22551 lands
 2022-02-07 14:06:19 -08:00
behzad nouri 27aaf9df85
removes VoteTracker::new in favor of VoteTracker::default (#22941) 
VoteTracker::new does not need a bank and is so redundant:
https://github.com/solana-labs/solana/blob/5a230f418/core/src/cluster_info_vote_listener.rs#L103-L107
 2022-02-04 19:01:59 +00:00
sakridge 5a230f418d
Add quic port for accepting transactions (#22753) 
using quinn library

streamer: Sign TLS cert with validator identity key

Handle multiple incoming chunks
 2022-02-04 15:27:09 +01:00
Tao Zhu 4bec182b32
Allow buffered packets be consumed if bank is active, regardless leader schedule (#22913) 2022-02-03 21:29:41 +00:00
Justin Starry 60af1a4cce
Refactor: Add trait for loading addresses (#22903) 2022-02-03 11:00:12 +00:00
carllin bd1850df25
Return actual committed transactions from process_transactions() (#22802) 2022-02-03 03:56:36 -05:00
Trent Nelson c62f9839a2 test-validator-bin: reinstate full rpc method set 2022-02-03 02:43:03 +00:00
Ikko Ashimine 58a70d76a3
fix typo in broadcast_duplicates_run.rs (#22888) 
Creat -> Create
 2022-02-02 12:29:14 -07:00
behzad nouri dccbddad80
adds reverse lookup index to cluster-nodes (#22892) 
retransmit has to exclude slot leader from set of nodes for each shred; 
which currently requires a linear scan:
https://github.com/solana-labs/solana/blob/e3b137066/core/src/cluster_nodes.rs#L238-L242

This commit adds a reverse lookup index to avoid linear scan.
 2022-02-02 19:27:50 +00:00
behzad nouri e3b137066d
caches WeightedShuffle struct in ClusterNodes (#22877) 
Instead of reconstructing WeightedShuffle struct for each shred
broadcast or retransmit, we can use the same struct with minimal
mutations.
 2022-02-02 15:12:26 +00:00
Trent Nelson eac4a6df68 rpc: use minimal mode by default 2022-02-01 19:00:06 -07:00
behzad nouri 45e09664b8
removes Rng field from WeightedShuffle struct (#22850) 2022-02-01 15:27:23 +00:00
behzad nouri 604ca9316c
includes zero weighted entries in WeightedShuffle (#22829) 
Current WeightedShuffle implementation excludes zero weighted entries
from the shuffle:
https://github.com/solana-labs/solana/blob/13e631dcf/gossip/src/weighted_shuffle.rs#L29-L30

Though mathematically this might make more sense, for our use-cases
(turbine specifically), this results in less efficient code:
https://github.com/solana-labs/solana/blob/13e631dcf/core/src/cluster_nodes.rs#L409-L430

This commit changes the implementation so that zero weighted indices are
also included in the shuffle but appear only at the end after non-zero
weighted indices.
 2022-01-31 16:23:50 +00:00
dependabot[bot] cc74693176
chore: bump serde from 1.0.134 to 1.0.136 (#22817) 
* chore: bump serde from 1.0.134 to 1.0.136

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.134 to 1.0.136.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.134...v1.0.136)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-01-28 12:20:37 -07:00
Justin Starry 220aa6ada0
Fix poh recorder initialization on startup (#22755) 2022-01-28 14:21:15 +08:00
Michael Vines 331b953551 Add vote account address to vote subscription 2022-01-27 08:22:29 -08:00
dependabot[bot] db481e1799
chore: bump serde_json from 1.0.75 to 1.0.78 (#22748) 
* chore: bump serde_json from 1.0.75 to 1.0.78

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.75 to 1.0.78.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.75...v1.0.78)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-01-26 18:01:02 -07:00
Justin Starry d9c259a231
Set the correct root in block commitment cache initialization (#22750) 
* Set the correct root in block commitment cache initialization

* clean up test

* bump
 2022-01-27 00:48:00 +08:00
sakridge 2e56c59bcb
Handle already discarded packets in gpu sigverify path (#22680) 2022-01-24 14:35:47 +01:00
Justin Starry 1240217a73
Refactor: Rename variables and helper method to `PohRecorder` (#22676) 
* Refactor: Rename leader_first_tick_height field

* Refactor: add `PohRecorder::slot_for_tick_height` helper

* Refactor: Add type for poh leader status
 2022-01-23 10:28:50 +08:00
anatoly yakovenko d6011ba14d
Dedup bloom filter is too slow (#22607) 
* Faster dedup

* use ahash

* fixup

* single threaded

* use duration type

* remove the count

* fixup
 2022-01-21 20:23:48 -07:00
Michael Vines 6d5bbca630 Pacify clippy 2022-01-21 19:12:57 -08:00
Michael Vines ce4f7601af Avoid unstable_name_collisions warning 2022-01-21 19:12:57 -08:00
dependabot[bot] 0cf886302d
chore: bump serde from 1.0.133 to 1.0.134 (#22650) 
* chore: bump serde from 1.0.133 to 1.0.134

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.133 to 1.0.134.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.133...v1.0.134)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-01-21 17:14:02 -07:00
dependabot[bot] 95bbb70c91
chore: bump serde_json from 1.0.74 to 1.0.75 (#22541) 
* chore: bump serde_json from 1.0.74 to 1.0.75

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.74 to 1.0.75.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.74...v1.0.75)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-01-21 12:26:51 -07:00
sakridge 38b02bbcc0
Handle already discarded packets in discard_excess_packets (#22594) 2022-01-21 17:22:50 +01:00
Jeff Biseda e7777281d6
regularly report network limits (#22563) 2022-01-20 12:38:42 -08:00
Trent Nelson cca3dbc76d system-monitor-service: support percentages from bigger numbers 2022-01-20 09:51:23 +00:00
Yueh-Hsuan Chiang f8db314134
(Ledger Store Benchmark) Display storage size of all data shreds (#22445) 
* (Ledger Store) APIs for obtaining physical size of all data and coding shreds

* (Ledger Store Benchmark) Display total data shred storage size.
 2022-01-19 19:33:08 -08:00
Justin Starry 7f20c6149e
Refactor: move simple vote parsing to runtime (#22537) 2022-01-20 10:39:21 +08:00
anatoly yakovenko d343713f61
Optimize packet dedup (#22571) 
* Use bloom filter to dedup packets

* dedup first

* Update bloom/src/bloom.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* fixup

* fixup

* fixup

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
 2022-01-19 13:58:20 -08:00
behzad nouri dcf44d2523
improves sigverify discard_excess_packets performance (#22577) 
As shown by the added benchmark, current code does worse if there is a
spam address plus a lot of unique addresses.

on current master:
test bench_packet_discard_many_senders  ... bench:   1,997,960 ns/iter (+/- 103,715)
test bench_packet_discard_mixed_senders ... bench:  14,256,116 ns/iter (+/- 534,865)
test bench_packet_discard_single_sender ... bench:   1,306,809 ns/iter (+/- 61,992)

with this commit:
test bench_packet_discard_many_senders  ... bench:   1,644,025 ns/iter (+/- 83,715)
test bench_packet_discard_mixed_senders ... bench:   1,089,789 ns/iter (+/- 86,324)
test bench_packet_discard_single_sender ... bench:     955,234 ns/iter (+/- 55,953)
 2022-01-19 18:10:02 +00:00
buffalu 650882217c
Add PacketBatch packet_indexes stat (#22564) 
* collect stats on packet batch indicies

* cleanup

* cleanup

* cleanup

* change name
 2022-01-19 08:13:07 +00:00
anatoly yakovenko e616a7ebfc
Track discard time of excess packets in sigverify (#22554) 
* discard time histogram

* closer to the if

* update
 2022-01-18 15:09:39 -07:00
Michael Vines 8dc6f9f589 Remove unused mut 2022-01-18 12:10:31 -08:00
sakridge 49443406fd
Use VecDeque instead of Vec in sigverify stage (#22538) 
avoid bad performance of remove(0) for a single sender
 2022-01-17 18:37:05 +01:00
anatoly yakovenko 2d94e6e5d3
metrics for generate new bank forks (#22492) 
* metrics for generate new bank forks

* fixed

* Apply suggestions from code review

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* --fixup

* fixup!

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
 2022-01-17 09:59:47 -07:00
Tao Zhu a724fa2347
Add hidden cli option to allow validator reports replayed transaction cost metrics (#22369) 
* add hidden cli option to allow validator reports replayed transaction cost detail metrics

* Update validator/src/main.rs

Co-authored-by: Michael Vines <mvines@gmail.com>

* - rebase master, using unbounded instead of channel; dowgrade to datapoint_trace

* removed cli arg, prefer log at trace

Co-authored-by: Michael Vines <mvines@gmail.com>
 2022-01-15 00:31:21 +00:00
Tao Zhu 1309a9cea0
Add estimated and actual block cost units metrics (#22326) 
* - report cost details for transactions selected to be packed into block;
- report estimated execution units packed into block, and actual units and time after execution

* revert reporting per-transaction details

* rollup transaction cost details (eg signature cost, wirte lock, data cost and execution costs) into block stats

* change naming from units to cu, use struct to replace tuple
 2022-01-14 23:44:18 +00:00
Tao Zhu 9c9f2dd5bd port counting vote CUs to block cost (#22477) 2022-01-14 10:50:29 -06:00
Justin Starry f804ccdece
Store address table lookups in blockstore and bigtable (#22402) 2022-01-14 15:24:41 +08:00
carllin 4ab7d6c23e
Filter out outdated slots (#22450) 
* Filter out outdated slots

* Fixup error
 2022-01-13 19:51:00 -05:00
Jeff Washington (jwash) e291342c4a
Revert "chore: bump dashmap from 4.0.2 to 5.0.0 (#21824)" (#22488) 
This reverts commit 8aa3d690b5.
 2022-01-13 13:06:39 -06:00
Tao Zhu 6614727be8 downgrade individual per-program-timing to trace to reduce writes to influx 2022-01-12 18:52:13 -06:00
Eric Warehime 8161cee70f
Remove unnecessary var in banking_stage bench (#22408) 2022-01-11 22:25:21 -06:00
Tyera Eulberg 637e366b18
Prevent rent-paying account creation (#22292) 
* Fixup typo

* Add new feature

* Add new TransactionError

* Add framework for checking account state before and after transaction processing

* Fail transactions that leave new rent-paying accounts

* Only check rent-state of writable tx accounts

* Review comments: combine process_result success behavior; log and metrics before feature activation

* Fix tests that assume rent-exempt accounts are okay

* Remove test no longer relevant

* Remove native/sysvar special case

* Move metrics submission to report legacy->legacy rent paying transitions as well
 2022-01-11 11:32:25 -07:00
Jeff Biseda 8b66625c95
convert std::sync::mpsc to crossbeam_channel (#22264) 2022-01-11 02:44:46 -08:00
dependabot[bot] 2cc6f863bf
Bump etcd-client from 0.8.2 to 0.8.3 (#22415) 
Bumps [etcd-client](https://github.com/etcdv3/etcd-client) from 0.8.2 to 0.8.3.
- [Release notes](https://github.com/etcdv3/etcd-client/releases)
- [Commits](https://github.com/etcdv3/etcd-client/compare/0.8.2...0.8.3)

---
updated-dependencies:
- dependency-name: etcd-client
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-10 13:25:15 -07:00
steviez 5f1f4dcbdd
Use struct to pass all Tpu sockets as one argument to Tpu::new() (#21965) 
Tpu::new() now matches Tvu::new() in having struct to reduce argument
list. Additionally, Rust supports partial moves, so there is no need to
clone the Tvu sockets out of Node object.
 2022-01-10 11:29:48 -06:00
Ashwin Sekar eeec1ce2ad
Add local cluster test to repro slot hash expiry bug (#21873) 2022-01-10 00:58:21 -05:00
dependabot[bot] 28275a33d6
Bump tempfile from 3.2.0 to 3.3.0 (#22401) 
* Bump tempfile from 3.2.0 to 3.3.0

Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases)
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.2.0...v3.3.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-01-09 12:26:08 -07:00
Yueh-Hsuan Chiang 5771c36d3f
Rename open_with_access_type() to open_with_options() (#22123) 2022-01-07 12:11:43 -08:00
Justin Starry 52d12cc802
Add runtime support for address table lookups (#22223) 
* Add support for address table lookups in runtime

* feedback

* feedback
 2022-01-07 11:59:09 +08:00
Trent Nelson 390ef0fbcd Consolidate process instruction execution timings to own struct 2022-01-06 03:56:46 -07:00
Trent Nelson 848b6dfbdd Add metrics for executor creation 2022-01-06 03:56:46 -07:00
Carl Lin b25e4a200b Add execute metrics 2022-01-06 03:56:46 -07:00
Trent Nelson 7d32909e17 move `ExecuteTimings` from `runtime::bank` to `program_runtime::timings` 2022-01-06 03:56:46 -07:00
Yueh-Hsuan Chiang 47b74e28ec
Add CLEANUP_SERVICE flag to ledger cleanup benchmark (#22108) 2022-01-05 23:46:02 -08:00
Yueh-Hsuan Chiang 44d61465f1
(Ledger store benchmark - 3/N) Add comments about the benchmark and its arguments (#22160) 
* Avoid shred generation in ledger_cleanup.rs

* Update comment for test_ledger_cleanup_compaction to include benchmark information.
 2022-01-04 23:35:55 -10:00
Justin Starry 45458e7139
Refactor: Improve type safety and readability of transaction execution (#22215) 
* Refactor Bank::load_and_execute_transactions

* Refactor: improve type safety of TransactionExecutionResult

* Add enum for extra type safety in execution results

* feedback
 2022-01-05 10:15:15 +08:00
Yueh-Hsuan Chiang e201b41341
Avoid shred generation in ledger_cleanup.rs (#22091) 2022-01-04 15:29:43 -10:00
behzad nouri 4b24499916 removes total-size from return value of recv_mmsg 2022-01-04 21:06:59 +00:00
behzad nouri 01a096adc8 adds bitflags to Packet.Meta 
Instead of a separate bool type for each flag, all the flags can be
encoded in a type-safe bitflags encoded in a single u8:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L19-L31
 2022-01-04 13:53:40 +00:00
behzad nouri 73a7741c49 uses std::net::IpAddr type for Packet.Meta.addr 2022-01-04 13:53:40 +00:00
sakridge 2486e21ffe
Lower vote-only-mode to 400 (#22210) 2022-01-04 12:49:14 +01:00
Jeff Biseda ca8fef5855
retransmit consecutive leader blocks (#22157) 2022-01-04 00:24:16 -08:00
Yueh-Hsuan Chiang e8b7f96a89
Add struct BlockstoreOptions (#22121) 2022-01-03 18:30:45 -10:00
dependabot[bot] 672fed04cb
Bump serde from 1.0.132 to 1.0.133 (#22233) 
* Bump serde from 1.0.132 to 1.0.133

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.132 to 1.0.133.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.132...v1.0.133)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-01-03 22:19:39 +00:00
carllin 005592998d
Fix bug, add error specific timings (#22225) 2022-01-03 16:33:54 -05:00
behzad nouri 69d71f8f86
removes epoch_authorized_voters from VoteTracker (#22192) 
https://github.com/solana-labs/solana/pull/22169
verifies authorized-voter early on in vote-listener pipeline; and so
VoteTracker no longer needs to maintain and check for epoch authorized
voters.
 2022-01-03 21:07:47 +00:00
Jeff Biseda 0e4ede46d1
work around rust 39364 for stats_reporter_sender (#22227) 2022-01-03 11:46:02 -08:00
dependabot[bot] ecbfc70bfa
Bump serde_json from 1.0.73 to 1.0.74 (#22231) 
* Bump serde_json from 1.0.73 to 1.0.74

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.73 to 1.0.74.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.73...v1.0.74)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2022-01-03 10:36:59 -07:00
Yueh-Hsuan Chiang 557d35ec79
Remove duplicate code in ledger_cleanup_compaction_test (#22204) 2021-12-31 11:19:33 -05:00
Yueh-Hsuan Chiang f479ab7af2
ledger_cleanup test improvement (1/N) -- make the test lockless and simplify the logic (#22090) 2021-12-30 20:18:47 -10:00
carllin d06e6c7425
Count compute units even when transaction errors (#22182) 2021-12-30 21:21:42 -05:00
Jeff Biseda 95dfcc546a
bypass retransmission for slots without propagated stats (#22176) 2021-12-30 16:07:34 -08:00
behzad nouri c0c6038654
checks for authorized voter early on in the vote-listener pipeline (#22169) 
Before votes are verified that they are signed by the authorized voter,
they might be dropped in verified-vote-packets code. If there are
enough many spam votes from unauthorized voters, this may potentially
drop valid votes but keep the false ones.
https://github.com/solana-labs/solana/blob/57986f982/core/src/verified_vote_packets.rs#L165-L168
 2021-12-30 15:03:14 +00:00
carllin 33d0b5e011
Revert "Count compute units even when transaction errors (#22059)" (#22174) 
This reverts commit eaa8c67bde.
 2021-12-30 02:42:32 -05:00
Lijun Wang f14928a970
Stream additional block metadata via plugin (#22023) 
* Stream additional block metadata through plugin
blockhash, block_height, block_time, rewards are streamed
 2021-12-29 15:12:01 -08:00
Justin Starry b1d9a2e60e
Don't forward packets received from TPU forwards port (#22078) 
* Don't forward packets received from TPU forwards port

* Add banking stage test
 2021-12-29 19:34:31 +01:00
dependabot[bot] d20a3774db
Bump lru from 0.7.1 to 0.7.2 (#22161) 
Bumps [lru](https://github.com/jeromefroe/lru-rs) from 0.7.1 to 0.7.2.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases)
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jeromefroe/lru-rs/compare/0.7.1...0.7.2)

---
updated-dependencies:
- dependency-name: lru
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-29 10:47:37 -07:00
carllin eaa8c67bde
Count compute units even when transaction errors (#22059) 2021-12-28 17:05:11 -05:00
carllin f061059e45
Prevent log spam (#22148) 2021-12-28 17:04:48 -05:00
Tao Zhu 3d6ab96587 push live packets straight to buffer, leader only process packets from buffer 2021-12-28 15:21:24 -06:00
Yueh-Hsuan Chiang b89cd8cd1a
Avoid cloning Vec<Entry> when calling entries_to_test_shreds() (#22093) 2021-12-24 12:32:43 -08:00
Justin Starry 93c776ce19
Refactor packet deduplication and harden bench test (#22080) 2021-12-22 23:05:10 -06:00
Tao Zhu dd80a525ef
Leader QoS service metrics (#21708) 
* - qos_service metrics tagged with leader thread ids to separate gossip/tpu votes and transactions;
- qos_service metrics is reported with bank slot;
- replaced timer-based reporting with signal via channel; removed async report test as qos_service now lives within a thread

* - add tpu live packets (eg, not buffered packets) states to qos metrics reporting
 2021-12-22 21:39:59 +00:00
behzad nouri f67ecd5c18
removes unused Cargo dependencies (#22022) 
Also moving some to [dev-dependencies] so that they are not propagated
to other packages which depend on the package.
 2021-12-22 17:01:33 +00:00
behzad nouri 4d62f03297
uses enum instead of trait for VoteTransaction (#22019) 
Box<dyn Trait> involves runtime dispatch, has significant overhead and
is slow. It also requires hacky boilerplate code for implementing Clone
or other basic traits:
https://github.com/solana-labs/solana/blob/e92a81b74/programs/vote/src/vote_state/mod.rs#L70-L102

Only limited known types can be VoteTransaction and they are all defined
in the same crate. So using a trait here only adds overhead.
https://github.com/solana-labs/solana/blob/e92a81b74/programs/vote/src/vote_state/mod.rs#L125-L165
https://github.com/solana-labs/solana/blob/e92a81b74/programs/vote/src/vote_state/mod.rs#L221-L264
 2021-12-22 14:25:46 +00:00
Tao Zhu 9c5d82557a skip reporting all-zero stats 2021-12-21 16:20:36 -06:00
dependabot[bot] 69d0b08dd8
chore: bump lru from 0.7.0 to 0.7.1 (#22018) 
Bumps [lru](https://github.com/jeromefroe/lru-rs) from 0.7.0 to 0.7.1.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases)
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jeromefroe/lru-rs/compare/0.7.0...0.7.1)

---
updated-dependencies:
- dependency-name: lru
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-20 21:56:56 -07:00
behzad nouri 65d59f4ef0
tracks erasure coding shreds' indices explicitly (#21822) 
The indices for erasure coding shreds are tied to data shreds:
https://github.com/solana-labs/solana/blob/90f41fd9b/ledger/src/shred.rs#L921

However with the upcoming changes to erasure schema, there will be more
erasure coding shreds than data shreds and we can no longer infer coding
shreds indices from data shreds.

The commit adds constructs to track coding shreds indices explicitly.
 2021-12-19 22:37:55 +00:00
behzad nouri 7476dfeec0
removes Select in favor of recv_timeout/try_iter (#21981) 
crossbeam_channel::Select::ready_timeout might return with success spuriously.
 2021-12-18 17:39:07 +00:00
Jeff Biseda 3fe942ab30
new net-stats require a new table (#21996) 2021-12-18 00:13:16 -08:00
dependabot[bot] 8f547a6c98
chore: bump serde from 1.0.131 to 1.0.132 (#21989) 
* chore: bump serde from 1.0.131 to 1.0.132

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.131 to 1.0.132.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.131...v1.0.132)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-12-18 07:45:20 +00:00
carllin 7f6fb6937a
Ensure AncestorHashesSerice selects an open port (#21919) 2021-12-18 00:44:01 -05:00
Jeff Biseda 97a1fa10a6
streamer send destination metrics for repair, gossip (#21564) 2021-12-17 15:21:05 -08:00
segfaultdoctor 76098dd42a
RPC Block Subscription (#21787) 
* add stuff

* compiling

* add notify block

* wip

* feat: add blockSubscribe pubsub method

* address PR comments

Co-authored-by: Lucas B <buffalu@jito.network>
Co-authored-by: Zano <segfaultdoctor@protonmail.com>
 2021-12-17 16:03:09 -07:00
behzad nouri 89d66c3210
removes next_shred_index from return value of entries to shreds api (#21961) 
next-shred-index is already readily available from returned data shreds.
The commit simplifies the api for upcoming changes to erasure coding
schema which will require explicit tracking of indices for coding shreds
as well as data shreds.
 2021-12-17 15:01:55 +00:00
Jeff Biseda 7ec39f5a1e
time based retransmit in replay_stage (#21498) 2021-12-17 05:44:40 -08:00
carllin 385efae4b3
Remove need to send bank in retransmit request from ReplayStage (#21943) 
* Remove need to send bank in retransmitter
 2021-12-16 21:11:01 -05:00
Justin Starry 6ff0be6a82
Clean up demote program write lock feature (#21949) 
* Clean up demote program write lock feature

* fix test
 2021-12-16 17:27:22 -05:00
steviez e83ca4bb28
Clean up test_ledger_cleanup_compaction prints (#21875) 
- Use info!()/warn!() over println!()/eprintln!()
- Make status prints consistent
- Add default RUST_LOG filter to see test printouts
- Adjust reported data to show shreds and rates we care about
 2021-12-16 11:24:29 -06:00
carllin cb395abff7
Fix subtraction overflow (#21871) 2021-12-14 14:24:22 -05:00
behzad nouri 8d980f07ba
uses Option<Slot> for SlotMeta.parent_slot (#21808) 
SlotMeta.parent_slot for the head of a detached chain of slots is
unknown and that is indicated by u64::MAX which lacks type-safety:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L203-L205

The commit changes the type to Option<Slot>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.
 2021-12-14 18:57:11 +00:00
behzad nouri 4ceb2689f5
adds ShredId uniquely identifying each shred (#21820) 2021-12-14 17:34:02 +00:00
steviez 4a9d7318d1
Rework test parameters to be shreds instead of entries (#21780) 
The number of shreds that result from a given number of entries is
variable and in our test case, somewhat unintuitive to think about when
trying to determine how much data we're pushing into the blockstore. So,
this change converts the unit of test parameters from entries to shreds.

This change also cleans up some variable naming for clarity and prints.
 2021-12-13 23:34:43 -06:00
dependabot[bot] 018b54dbd7
chore: bump serde_json from 1.0.72 to 1.0.73 (#21856) 
* chore: bump serde_json from 1.0.72 to 1.0.73

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.72 to 1.0.73.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.72...v1.0.73)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-12-14 04:08:22 +00:00
dependabot[bot] b81124deec
chore: bump etcd-client from 0.8.1 to 0.8.2 (#21825) 
Bumps [etcd-client](https://github.com/etcdv3/etcd-client) from 0.8.1 to 0.8.2.
- [Release notes](https://github.com/etcdv3/etcd-client/releases)
- [Commits](https://github.com/etcdv3/etcd-client/compare/0.8.1...0.8.2)

---
updated-dependencies:
- dependency-name: etcd-client
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-13 16:20:22 -07:00
dependabot[bot] 8aa3d690b5
chore: bump dashmap from 4.0.2 to 5.0.0 (#21824) 
* chore: bump dashmap from 4.0.2 to 5.0.0

Bumps [dashmap](https://github.com/xacrimon/dashmap) from 4.0.2 to 5.0.0.
- [Release notes](https://github.com/xacrimon/dashmap/releases)
- [Commits](https://github.com/xacrimon/dashmap/commits/v5.0.0)

---
updated-dependencies:
- dependency-name: dashmap
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-12-13 13:23:58 -07:00
Jeff Washington (jwash) 90f41fd9b7
use cost model to limit new account creation (#21369) 
* use cost model to limit new account creation

* handle every system instruction

* remove &

* simplify match

* simplify match

* add datapoint for account data size

* add postgres error handling

* handle accounts:unlock_accounts
 2021-12-12 14:57:18 -06:00
behzad nouri e08139f949
uses Option<u64> for SlotMeta.last_index (#21775) 
SlotMeta.last_index may be unknown and current code is using u64::MAX to
indicate that:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L169-L174

This lacks type-safety and can introduce bugs if not always checked for
Several instances of slot_meta.last_index + 1 are also subject to
overflow.

This commit updates the type to Option<u64>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.
 2021-12-11 14:47:20 +00:00
Justin Starry 254ef3e7b6
Rename Packets to PacketBatch (#21794) 2021-12-11 09:44:15 -05:00
Yueh-Hsuan Chiang 65194c7ae8
Add NUM_WRITERS to ledger_cleanup to enable multiple writers. (#21729) 
Summary:
* Add NUM_WRITERS to ledger_cleanup to enable multiple writers.
  (Note that our insert_shreds() is still single threaded because
   it has a lock that limits only one writer at a time.)

* Make pre-generated slots more performent by directly inserting
  into the shared queue.  Otherwise, the main-thread which
  prepares the slots will be slower than the writers.

* Correct the shred insertion time -- before this diff it did not
  wait for joining all writer threads.
 2021-12-10 09:42:51 -08:00
dependabot[bot] d870f566ef
chore: bump serde from 1.0.130 to 1.0.131 (#21758) 
* chore: bump serde from 1.0.130 to 1.0.131

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.130 to 1.0.131.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.130...v1.0.131)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-12-10 08:47:46 +00:00
behzad nouri 8063273d09
adds more sanity checks to shreds (#21675) 2021-12-09 16:43:57 +00:00
Ashwin Sekar f0acf7681e
Add vote instructions that directly update on chain vote state (#21531) 
* Add vote state instructions

UpdateVoteState and UpdateVoteStateSwitch

* cargo tree

* extract vote state version conversion to common fn
 2021-12-07 16:47:26 -08:00
dependabot[bot] f6801a4af4
chore: bump itertools from 0.10.1 to 0.10.3 (#21643) 
* chore: bump itertools from 0.10.1 to 0.10.3

Bumps [itertools](https://github.com/rust-itertools/itertools) from 0.10.1 to 0.10.3.
- [Release notes](https://github.com/rust-itertools/itertools/releases)
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.10.1...v0.10.3)

---
updated-dependencies:
- dependency-name: itertools
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-12-06 23:23:26 -07:00
behzad nouri cd17f63d81
adds back position field to coding-shred-header (#21600) 
https://github.com/solana-labs/solana/pull/17004
removed position field from coding-shred-header because as it stands the
field is redundant and unused.
However, with the upcoming changes to erasure coding schema this field
will no longer be redundant and needs to be populated.
 2021-12-05 14:42:09 +00:00
Trent Nelson d6f22433d0 Bump version to v1.10.0 2021-12-04 20:17:54 +00:00
Jeff Biseda 9c6b95e1e1
fix distance calculation in get_closest_completion (#21601) 2021-12-03 22:36:46 -08:00
dependabot[bot] 446089edb4
chore: bump retain_mut from 0.1.4 to 0.1.5 (#21599) 
Bumps [retain_mut](https://github.com/upsuper/retain_mut) from 0.1.4 to 0.1.5.
- [Release notes](https://github.com/upsuper/retain_mut/releases)
- [Commits](https://github.com/upsuper/retain_mut/compare/v0.1.4...v0.1.5)

---
updated-dependencies:
- dependency-name: retain_mut
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-03 22:11:42 -07:00
Justin Starry 1430b58a6d
Remove deprecated slow epoch boundary methods (#21568) 2021-12-03 17:59:10 +00:00
Michael Vines b8837c04ec Reformat imports to a consistent style for imports 
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
 2021-12-03 09:19:13 -08:00
Alexander Meißner b78f5b6032
Refactor: Cleanup InstructionProcessor (#21404) 
* Moves create_message(), native_invoke() and process_cross_program_instruction()
from the InstructionProcessor to the InvokeContext so that they can have a useful "self" parameter.

* Moves InstructionProcessor into InvokeContext and Bank.

* Moves ExecuteDetailsTimings into its own file.

* Moves Executor into invoke_context.rs

* Moves PreAccount into its own file.

* impl AbiExample for BuiltinPrograms
 2021-12-01 08:54:42 +01:00
Michael Vines dd12d90eac Upgrade to Rust 2021 2021-11-30 20:43:46 -08:00
dependabot[bot] de8edad30b
chore: bump etcd-client from 0.7.2 to 0.8.1 (#21513) 
Bumps [etcd-client](https://github.com/etcdv3/etcd-client) from 0.7.2 to 0.8.1.
- [Release notes](https://github.com/etcdv3/etcd-client/releases)
- [Commits](https://github.com/etcdv3/etcd-client/compare/0.7.2...0.8.1)

---
updated-dependencies:
- dependency-name: etcd-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-30 18:16:32 +00:00
dependabot[bot] a125e7b8c2
chore: bump serde_json from 1.0.71 to 1.0.72 (#21426) 
* chore: bump serde_json from 1.0.71 to 1.0.72

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.71 to 1.0.72.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.71...v1.0.72)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-11-30 18:03:09 +00:00
Michael Vines ba9dfa0d22 Remove frozen account support 2021-11-29 08:38:11 -08:00
Tao Zhu 9edfc5936d
Refactor accounts.rs with Justin's comments to improve lock accounts (#21406) 
with results code path.
- fix a bug that could unlock accounts that weren't locked
- add test to the refactored function
- skip enumerating transaction accounts if qos results is an error
- add #[must_use] annotation
- avoid clone error in results
- add qos error code to unlock_accounts match statement
- remove unnecessary AbiExample
 2021-11-23 21:17:55 +00:00
Lijun Wang c29838fce1
Accountsdb plugin transaction part 3: Transaction Notifier (#21374) 
The TransactionNotifierInterface interface for notifying transactions.
Changes to transaction_status_service to notify the notifier of the transaction data.
Interface to query the plugin's interest in transaction data
 2021-11-23 09:55:53 -08:00
Tao Zhu 2602e7c3bc
Fix flaky test (#21402) 
* the async test is flaky on ci

* fix unstable test by increasing stats repoting time
 2021-11-23 09:47:17 -06:00
behzad nouri dd338b6c9f
changes Shred::parent return type to Option<Slot> (#21370) 
Shred::parent can return garbage if the struct fields are invalid:
https://github.com/solana-labs/solana/blob/8a50b6302/ledger/src/shred.rs#L446-L453

The commit adds more sanity checks and changes the return type to Option<Slot>.
 2021-11-23 14:45:26 +00:00
Tao Zhu cd5a39ee43
the async test is flaky on ci (#21365) 2021-11-22 18:16:20 -06:00
Jeff Washington (jwash) 87831e7f8d
start system monitor earlier in validator so we get memory stats at startup (#21372) 2021-11-22 14:37:17 -06:00
sakridge f31ca8ba8c
Report cluster slots size (#21380) 2021-11-22 17:47:58 +01:00
Jeff Biseda 2ed7e3af89
prioritize slot repairs for unknown last index and close to completion (#21070) 2021-11-19 19:17:30 -08:00
sakridge 0bda0c3e0c
Add bank drop service (#21322) 2021-11-19 17:20:18 +01:00
behzad nouri 48dfdfb4d5 changes Blockstore::is_shred_duplicate arg type to ShredType 2021-11-19 14:16:39 +00:00
behzad nouri 57057f8d39 uses enum for shred type 
Current code is using u8 which does not have any type-safety and can
contain invalid values:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L167

Checks for invalid shred-types are scattered through the code:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/blockstore.rs#L849-L851
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L346-L348

The commit uses enum for shred type with #[repr(u8)]. Backward
compatibility is maintained by implementing Serialize and Deserialize
compatible with u8, and adding a test to assert that.
 2021-11-19 14:16:39 +00:00