bab96e878e
Revert "chore: bump trees from 0.2.1 to 0.4.2 ( #18041 )" ( #18045 )
...
This reverts commit f2a7152c0c
2021-06-17 14:04:59 -05:00
071b1ee3e5
Removed pub from some functions which are actually private to improve encapsulation ( #18030 )
...
Remove the pub marker to improve encapsulation. Readability improvement only, no functional impact.
2021-06-17 10:14:21 -07:00
f2a7152c0c
chore: bump trees from 0.2.1 to 0.4.2 ( #18041 )
...
Bumps [trees](https://github.com/oooutlk/trees ) from 0.2.1 to 0.4.2.
- [Release notes](https://github.com/oooutlk/trees/releases )
- [Commits](https://github.com/oooutlk/trees/commits )
---
updated-dependencies:
- dependency-name: trees
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-17 08:07:16 -06:00
fa04531c7a
Extricate RpcCompletedSlotsService from RetransmitStage
2021-06-16 16:20:35 -07:00
5bc6c89adc
validator: run poh speed test earlier in start up
2021-06-16 21:27:08 +00:00
6f5b248746
chore: bump miow from 0.2.2 to 0.3.6 ( #18000 )
...
* chore: bump miow from 0.2.2 to 0.3.6
Bumps [miow](https://github.com/yoshuawuyts/miow ) from 0.2.2 to 0.3.6.
- [Release notes](https://github.com/yoshuawuyts/miow/releases )
- [Changelog](https://github.com/yoshuawuyts/miow/blob/master/CHANGELOG.md )
- [Commits](https://github.com/yoshuawuyts/miow/compare/0.2.2...v0.3.6 )
---
updated-dependencies:
- dependency-name: miow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-06-16 12:30:17 -06:00
dbd4dc04b0
ledger tool limit_load_slot_count_from_snapshot avoids assert failures ( #17974 )
2021-06-15 15:39:22 -05:00
f558b9b6bf
verify bank hash on startup with ledger tool option ( #17939 )
2021-06-15 11:52:12 -05:00
1b1d34da59
Refactor stake program into solana_program ( #17906 )
...
* Move stake state / instructions into solana_program
* Update account-decoder
* Update cli and runtime
* Update all other parts
* Commit Cargo.lock changes in programs/bpf
* Update cli stake instruction import
* Allow integer arithmetic
* Update ABI digest
* Bump rust mem instruction count
* Remove useless structs
* Move stake::id() -> stake::program::id()
* Re-export from solana_sdk and mark deprecated
* Address feedback
* Run cargo fmt
2021-06-15 18:04:00 +02:00
161838655c
removes port-based forwarding logic from turbine retransmit ( #17716 )
...
Turbine retransmit logic is based on which socket it received the packet
from (i.e `packet.meta.forward`):
https://github.com/solana-labs/solana/blob/708bbcb00/core/src/retransmit_stage.rs#L467-L470
This can leave the cluster vulnerable to spoofing and selective
propagation of packets; see
https://github.com/solana-labs/solana/issues/6672
https://github.com/solana-labs/solana/pull/7774
This commit identifies if the node is on the "critical path" based on
its index in the shuffled cluster. If so, it forwards the packet to both
neighbors and children; otherwise, the packet is only forwarded to the
children.
The metrics added in
https://github.com/solana-labs/solana/pull/17351
shows that the number of times the index does not match the port is very
rare, and therefore this change should be safe.
2021-06-15 13:19:41 +00:00
ccc013e134
Handle removing slots during account scans ( #17471 )
2021-06-14 21:04:01 -07:00
471b34132e
add metrics for startup ( #17913 )
...
* add metrics for startup
* roll timings up higher
* fix test
* fix duplicate
2021-06-14 17:46:49 -05:00
e6bbd4b3f0
add metrics to handle_snapshot_requests ( #17937 )
2021-06-14 15:46:19 -05:00
a0872232d3
chore: bump itertools from 0.9.0 to 0.10.1 ( #17929 )
...
* chore: bump itertools from 0.9.0 to 0.10.1
Bumps [itertools](https://github.com/rust-itertools/itertools ) from 0.9.0 to 0.10.1.
- [Release notes](https://github.com/rust-itertools/itertools/releases )
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.9.0...v0.10.1 )
---
updated-dependencies:
- dependency-name: itertools
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Fix versions
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-06-14 18:32:20 +00:00
a248770a6b
chore: bump systemstat from 0.1.7 to 0.1.8 ( #17928 )
...
Bumps [systemstat](https://github.com/unrelentingtech/systemstat ) from 0.1.7 to 0.1.8.
- [Release notes](https://github.com/unrelentingtech/systemstat/releases )
- [Commits](https://github.com/unrelentingtech/systemstat/compare/v0.1.7...v0.1.8 )
---
updated-dependencies:
- dependency-name: systemstat
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-14 16:48:54 +00:00
eeee75c5be
Don't use pinned memory when unnecessary ( #17832 )
...
Reports of excessive GPU memory usage and errors
from cudaHostRegister. There are some cases where pinning is
not required.
2021-06-14 16:10:04 +02:00
a501707058
chore: bump ahash from 0.6.1 to 0.7.4 ( #17886 )
...
Bumps [ahash](https://github.com/tkaitchuck/ahash ) from 0.6.1 to 0.7.4.
- [Release notes](https://github.com/tkaitchuck/ahash/releases )
- [Commits](https://github.com/tkaitchuck/ahash/commits )
---
updated-dependencies:
- dependency-name: ahash
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-11 14:25:25 -06:00
0feac57cb0
Don't store votes unless we are leader soon ( #17803 )
2021-06-11 18:29:05 +02:00
2aa7df23b5
chore: bump indexmap from 1.5.1 to 1.6.2 ( #17884 )
...
* chore: bump indexmap from 1.5.1 to 1.6.2
Bumps [indexmap](https://github.com/bluss/indexmap ) from 1.5.1 to 1.6.2.
- [Release notes](https://github.com/bluss/indexmap/releases )
- [Commits](https://github.com/bluss/indexmap/compare/1.5.1...1.6.2 )
---
updated-dependencies:
- dependency-name: indexmap
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-06-11 09:50:09 -06:00
c8535be0e1
Port unconfirmed duplicate tracking logic from ProgressMap to ForkChoice ( #17779 )
2021-06-11 03:09:57 -07:00
afafa624a3
Account for duplicate before a bank is frozen or replayed ( #17866 )
2021-06-10 22:28:23 -07:00
f08ed1eb2d
chore: bump rayon from 1.5.0 to 1.5.1 ( #17869 )
...
* chore: bump rayon from 1.5.0 to 1.5.1
Bumps [rayon](https://github.com/rayon-rs/rayon ) from 1.5.0 to 1.5.1.
- [Release notes](https://github.com/rayon-rs/rayon/releases )
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md )
- [Commits](https://github.com/rayon-rs/rayon/compare/rayon-core-v1.5.0...v1.5.1 )
---
updated-dependencies:
- dependency-name: rayon
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-06-10 13:18:47 -06:00
269d995832
Make account shrink configurable #17544 ( #17778 )
...
1. Added both options for measuring space usage using total accounts usage and for individual store shrink ratio using an enum. Validator CLI options: --accounts-shrink-optimize-total-space and --accounts-shrink-ratio
2. Added code for selecting candidates based on total usage in a separate function select_candidates_by_total_usage
3. Added unit tests for the new functions added
4. The default implementations is kept at 0.8 shrink ratio with --accounts-shrink-optimize-total-space set to true
Fixes #17544
2021-06-09 21:21:32 -07:00
ae27fcbcda
replay stage feed back program cost ( #17731 )
...
* replay stage feeds back realtime per-program execution cost to cost model;
* program cost execution table is initialized into empty table, no longer populated with hardcoded numbers;
* changed cost unit to microsecond, using value collected from mainnet;
* add ExecuteCostTable with fixed capacity for security concern, when its limit is reached, programs with old age AND less occurrence will be pushed out to make room for new programs.
2021-06-09 17:10:59 -05:00
050bb5446d
Add local cluster tests that broadcast duplicate slots ( #13995 )
...
* Add duplicate node local cluster test
* fix clippy
* remove dupe test
2021-06-09 15:01:48 -07:00
9a2ca8dd2f
chore: bump rustc_version from 0.2.3 to 0.4.0 ( #17854 )
...
* chore: bump rustc_version from 0.2.3 to 0.4.0
Bumps [rustc_version](https://github.com/Kimundi/rustc-version-rs ) from 0.2.3 to 0.4.0.
- [Release notes](https://github.com/Kimundi/rustc-version-rs/releases )
- [Commits](https://github.com/Kimundi/rustc-version-rs/compare/v0.2.3...v0.4.0 )
---
updated-dependencies:
- dependency-name: rustc_version
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Make versions consistent
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-06-09 16:57:39 +00:00
e5e7390d44
Wrap long lines
2021-06-08 12:05:29 -07:00
2e998ed11d
Bump jsonrpc crates ( #17797 )
...
* Bump jsonrpc crates
* Update error text
* gitignore for rpc
2021-06-07 18:03:20 +00:00
8a77b21be1
chore: bump retain_mut from 0.1.2 to 0.1.3 ( #17736 )
...
Bumps [retain_mut](https://github.com/upsuper/retain_mut ) from 0.1.2 to 0.1.3.
- [Release notes](https://github.com/upsuper/retain_mut/releases )
- [Commits](https://github.com/upsuper/retain_mut/compare/v0.1.2...v0.1.3 )
---
updated-dependencies:
- dependency-name: retain_mut
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-04 17:36:21 +00:00
544b3c0d17
Create solana-poh and move remaining rpc modules to solana-rpc ( #17698 )
...
* Create solana-poh crate
* Move BigTableUploadService to solana-ledger
* Add solana-rpc to workspace
* Move dependencies to solana-rpc
* Move remaining rpc modules to solana-rpc
* Single use statement solana-poh
* Single use statement solana-rpc
2021-06-04 09:23:06 -06:00
f97ce2cd7e
Per-program id timings ( #17554 )
2021-06-04 16:04:31 +02:00
be957f25c9
adds fallback logic if retransmit multicast fails ( #17714 )
...
In retransmit-stage, based on the packet.meta.seed and resulting
children/neighbors, each packet is sent to a different set of peers:
https://github.com/solana-labs/solana/blob/708bbcb00/core/src/retransmit_stage.rs#L421-L457
However, current code errors out as soon as a multicast call fails,
which will skip all the remaining packets:
https://github.com/solana-labs/solana/blob/708bbcb00/core/src/retransmit_stage.rs#L467-L470
This can exacerbate packets loss in turbine.
This commit:
* keeps iterating over retransmit packets for loop even if some
intermediate sends fail.
* adds a fallback to UdpSocket::send_to if multicast fails.
Recent discord chat:
https://discord.com/channels/428295358100013066/689412830075551748/849530845052403733
2021-06-04 12:16:37 +00:00
3a647c4bea
Rename ValidatorExit and move to sdk ( #17728 )
2021-06-04 03:06:13 +00:00
3670435db4
chore: bump serial_test from 0.4.0 to 0.5.1 ( #17705 )
...
Bumps [serial_test](https://github.com/palfrey/serial_test ) from 0.4.0 to 0.5.1.
- [Release notes](https://github.com/palfrey/serial_test/releases )
- [Commits](https://github.com/palfrey/serial_test/compare/v0.4.0...v0.5.1 )
---
updated-dependencies:
- dependency-name: serial_test
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-03 10:07:56 -06:00
96ba2edfeb
Switch EpochSlots to be frozen slots, not completed slots ( #17168 )
2021-06-03 00:20:00 +00:00
19c49e2e25
chore: bump crossbeam-channel from 0.4.4 to 0.5.0 ( #17665 )
...
* chore: bump crossbeam-channel from 0.4.4 to 0.5.0
Bumps [crossbeam-channel](https://github.com/crossbeam-rs/crossbeam ) from 0.4.4 to 0.5.0.
- [Release notes](https://github.com/crossbeam-rs/crossbeam/releases )
- [Changelog](https://github.com/crossbeam-rs/crossbeam/blob/master/CHANGELOG.md )
- [Commits](https://github.com/crossbeam-rs/crossbeam/compare/crossbeam-channel-0.4.4...crossbeam-channel-0.5.0 )
---
updated-dependencies:
- dependency-name: crossbeam-channel
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-06-02 19:36:28 +00:00
d99c888cc2
chore: bump systemstat from 0.1.5 to 0.1.7 ( #17667 )
...
Bumps [systemstat](https://github.com/myfreeweb/systemstat ) from 0.1.5 to 0.1.7.
- [Release notes](https://github.com/myfreeweb/systemstat/releases )
- [Commits](https://github.com/myfreeweb/systemstat/commits/v0.1.7 )
---
updated-dependencies:
- dependency-name: systemstat
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-02 09:08:07 -06:00
bbcdf073ba
Support out of band dumping of unrooted slots in AccountsDb ( #17269 )
...
* Accounts dumping logic
* Add test for interaction between cache flush and remove_unrooted_slot()
* Update comments
* Rename
* renaming
* Add more comments
* Renaming
* Fixup test and bad check
2021-06-02 09:51:10 +00:00
b000d490ce
Cost Model to limit transactions which are not parallelizeable ( #16694 )
...
* * Add following to banking_stage:
1. CostModel as immutable ref shared between threads, to provide estimated cost for transactions.
2. CostTracker which is shared between threads, tracks transaction costs for each block.
* replace hard coded program ID with id() calls
* Add Account Access Cost as part of TransactionCost. Account Access cost are weighted differently between read and write, signed and non-signed.
* Establish instruction_execution_cost_table, add function to update or insert instruction cost, unit tested. It is read-only for now; it allows Replay to insert realtime instruction execution costs to the table.
* add test for cost_tracker atomically try_add operation, serves as safety guard for future changes
* check cost against local copy of cost_tracker, return transactions that would exceed limit as unprocessed transaction to be buffered; only apply bank processed transactions cost to tracker;
* bencher to new banking_stage with max cost limit to allow cost model being hit consistently during bench iterations
2021-06-01 09:16:17 -05:00
ab0f4ff835
Bump serde from 1.0.122 to 1.0.126 ( #17618 )
...
* Bump serde from 1.0.122 to 1.0.126
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.122 to 1.0.126.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.122...v1.0.126 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2021-05-31 22:41:25 +00:00
ac452f9bb8
Bump tempfile from 3.1.0 to 3.2.0 ( #17587 )
...
* Bump tempfile from 3.1.0 to 3.2.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.1.0 to 3.2.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases )
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS )
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.1.0...v3.2.0 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-05-31 18:05:07 +00:00
86c07bf785
Bump bs58 from 0.3.1 to 0.4.0 ( #17573 )
...
* Bump bs58 from 0.3.1 to 0.4.0
Bumps [bs58](https://github.com/mycorrhiza/bs58-rs ) from 0.3.1 to 0.4.0.
- [Release notes](https://github.com/mycorrhiza/bs58-rs/releases )
- [Commits](https://github.com/mycorrhiza/bs58-rs/compare/0.3.1...0.4.0 )
Signed-off-by: dependabot[bot] <support@github.com>
* Fixup
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-05-28 18:30:34 +00:00
1f97b2365f
Avoid full-range compactions with periodic filtered b.g. ones ( #16697 )
...
* Update rocksdb to v0.16.0
* Promote the infrequent and important log to info!
* Force background compaction by ttl without manual compaction
* Fix test
* Support no compaction mode in test_ledger_cleanup_compaction
* Fix comment
* Make compaction_interval customizable
* Avoid major compaction with periodic filtering...
* Adress lazy_static, special cfs and range check
* Clean up a bit and add comment
* Add comment
* More comments...
* Config code cleanup
* Add comment
* Use .conflicts_with()
* Nullify unneeded delete_range ops for special CFs
* Some clean ups
* Clarify the locking intention
* Ensure special CFs' consistency with PurgeType::CompactionFilter
* Fix comment
* Fix bad copy paste
* Fix various types...
* Don't use tuples
* Add a unit test for compaction_filter
* Fix typo...
* Remove flag and just use new behavior always
* Fix wrong condition negation...
* Doc. about no set_last_purged_slot in purge_slots
* Write a test and fix off-by-one bug....
* Apply suggestions from code review
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Follow up to github review suggestions
* Fix line-wrapping
* Fix conflict
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-05-28 16:42:56 +09:00
8eab0e8602
Bump version to v1.8.0 ( #17541 )
2021-05-27 08:51:53 -07:00
ab581dafc2
Add block height to ConfirmedBlock structs ( #17523 )
...
* Add BlockHeight CF to blockstore
* Rename CacheBlockTimeService to be more general
* Cache block-height using service
* Fixup previous proto mishandling
* Add block_height to block structs
* Add block-height to solana block
* Fallback to BankForks if block time or block height are not yet written to Blockstore
* Add docs
* Review comments
2021-05-26 22:16:16 -06:00
9541411c15
Plumb transaction-level rewards (aka "rent debits") into the `getTransaction` RPC method
2021-05-27 03:05:05 +00:00
52dccc656a
Purge slots greater than new last index ( #16071 )
2021-05-26 16:12:57 -07:00
cbce440af4
simulateTransaction can now return accounts modified by the simulation
2021-05-26 14:20:23 -07:00
6abe089740
Add custom error for tx-history queries when node does not support ( #17494 )
2021-05-26 13:27:41 -06:00
6b9d8d41a3
add --limit_load_slot_count_from_snapshot to ledger-tool ( #17417 )
2021-05-26 10:36:12 -05:00
9a5330b7eb
Move gossip modules into solana-gossip crate ( #17352 )
...
* Move gossip modules to solana-gossip
* Update Protocol abi digest due to move
* Move gossip benches and hook up CI
* Remove unneeded Result entries
* Single use statements
2021-05-26 09:15:46 -06:00
e9bc1c6b07
Add last valid block height to rpc Fees ( #17506 )
...
* Add last_valid_block_height to fees rpc
* Add getBlockHeight rpc
* Update docs
2021-05-26 07:26:19 +00:00
660d37aadf
sigVerify conflicts with replace, add tests
2021-05-25 17:32:00 -07:00
e14f3eb529
rename flag
2021-05-25 17:32:00 -07:00
96cef5260c
Add a flag to simulateTransaction to use most recent blockhash
2021-05-25 17:32:00 -07:00
d8bc56fa51
Refactor purge_slots_from_cache_and_store() and handle_reclaims() ( #17319 )
2021-05-24 13:51:17 -07:00
41ec1c8d50
Add blockstore-root-scan for api nodes on boot ( #17402 )
...
* Add blockstore-root-scan for api nodes on boot
* Ensure cluster-confirmed root and parents are set as root in blockstore in load_frozen_forks()
* Plumb rpc-scan-and-fix-roots validator flag
2021-05-24 13:24:47 -06:00
30b60a976b
Avoid ip_echo_server unwrap
2021-05-24 12:10:50 -07:00
e867d7f3b8
removes Crds::lookup and lookup_versioned ( #17438 )
2021-05-24 18:21:54 +00:00
a8dca3976b
Refactor genesis download/load/check functions ( #17276 )
...
* Refactor genesis ingest functions
* Consolidate genesis.bin/genesis.tar.bz2 references
2021-05-24 16:45:36 +02:00
9d112cf41f
encapsulates purged values bookkeeping into crds module ( #17265 )
...
For all code paths (gossip push, pull, purge, etc) that remove or
override a crds value, it is necessary to record hash of values purged
from crds table, in order to exclude them from subsequent pull-requests;
otherwise the next pull request will likely return outdated values,
wasting bandwidth:
https://github.com/solana-labs/solana/blob/ed51cde37/core/src/crds_gossip_pull.rs#L486-L491
Currently this is done all over the place in multiple modules, and this
has caused bugs in the past where purged values were not recorded.
This commit encapsulated this bookkeeping into crds module, so that any
code path which removes or overrides a crds value, also records the hash
of purged value in-place.
2021-05-24 13:47:21 +00:00
060332c704
indexes crds votes by insert order ( #17340 )
...
Crds::get_votes is scanning over all votes in the crds table only to
return those inserted since the given cursor:
https://github.com/solana-labs/solana/blob/2ae57c172/core/src/crds.rs#L250-L266
Having votes indexed by insert order avoids the table scan and will be
more efficient.
2021-05-24 13:35:01 +00:00
5567305a5f
rolls back min number of bloom items for debug builds ( #17420 )
...
coverage ci builds are have become flaky presumably because of the
overhead added in https://github.com/solana-labs/solana/pull/17236
for very small test clusters.
This commit uses a smaller min number of bloom items condition on that
if debug assertions are enabled or not.
Previous attempt at fixing the flakiness:
https://github.com/solana-labs/solana/pull/17408
2021-05-23 16:50:19 +00:00
8664b2cc39
Fix bad assertion ( #17401 )
2021-05-22 20:18:13 -07:00
cf1acfb021
uses Duration type for gossip discover timeout
2021-05-22 19:17:36 +00:00
d6496376ce
increases timeout duration for gossip discover
2021-05-22 19:17:36 +00:00
a7870cda8d
records hash of timed-out pull responses
...
Gossip should record hash of pull responses which are timed out and
fail to insert:
https://github.com/solana-labs/solana/blob/ed51cde37/core/src/crds_gossip_pull.rs#L397-L400
so that they are excluded from the next pull request:
https://github.com/solana-labs/solana/blob/ed51cde37/core/src/crds_gossip_pull.rs#L486-L491
otherwise the next pull request will likely include the same timed out
values and waste bandwidth.
2021-05-22 17:02:24 +00:00
d41266e4e9
rpc: add context toggle to getProgramAccounts ( #17399 )
...
* fix(rpc): return context in get_program_accounts
* doc(rpc): document withContext flag
* fix(rpc): fix comment
Co-authored-by: Michael Vines <mvines@gmail.com>
* fix(rpc): fix doc
Co-authored-by: Michael Vines <mvines@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-05-22 07:12:21 +00:00
9471ba61c5
removes redundant slots sort in push_epoch_slots
2021-05-21 19:17:15 +00:00
9339a6f8f3
locks gossip only once in push_epoch_slots
...
push_epoch_slots is unlocking and locking again gossip when iterating
over epoch slot indices which is wasteful:
https://github.com/solana-labs/solana/blob/0486df02b/core/src/cluster_info.rs#L915-L929
2021-05-21 19:17:15 +00:00
ff0e623d30
removes the nested for loop from retransmit-stage
...
The code can be simplified by just flattening the vector of packets.
2021-05-21 17:10:56 +00:00
71de021177
adds metric for turbine retransmit tree mismatch
...
In order to remove port-based forwarding logic in turbine, we need to
first track how often the turbine retransmit/broadcast trees mismatch
across nodes.
One consistency condition is that if the node is on the critical path
(i.e. the first node in each neighborhood), then we expect that the
packet arrives at tvu socket as opposed to tvu-forwards.
This commit adds a metric to track how often above condition is not met.
2021-05-21 17:10:56 +00:00
2adce67260
extends crds values timeouts if stakes are unknown ( #17261 )
...
If stakes are unknown, then timeouts will be short, resulting in values
being purged from the crds table, and consequently higher pull-response
load when they are obtained again from gossip. In particular, this slows
down validator start where almost all values obtained from entrypoint
are immediately discarded.
2021-05-21 15:55:22 +00:00
5e6b00fe98
prioritizes more recent values in pull responses ( #17238 )
...
On the receiving end, the outdated values are discarded, and they will
only waste bandwidth:
https://github.com/solana-labs/solana/blob/3f0480d06/core/src/crds_gossip_pull.rs#L385-L400
This is also exacerbating validator start, since the entrypoint is
returning old values in pull responses, and the validator immediately
discards those; resulting in huge delay until the validator obtains
contact-info of the entrypoint and is able to adopt shred-version and
fully start.
2021-05-21 14:07:46 +00:00
e8b35a4f7b
bumps up min number of bloom items in gossip pull requests ( #17236 )
...
When a validator starts, it has an (almost) empty crds table and it only
sends one pull-request to the entrypoint. The bloom filter in the
pull-request targets 10% false rate given the number of items. So, if
the `num_items` is very wrong, it makes a very small bloom filter with a
very high false rate:
https://github.com/solana-labs/solana/blob/2ae57c172/runtime/src/bloom.rs#L70-L80
https://github.com/solana-labs/solana/blob/2ae57c172/core/src/crds_gossip_pull.rs#L48
As a result, it is very unlikely that the validator obtains entrypoint's
contact-info in response. This exacerbates how long the validator will
loop on:
> Waiting to adopt entrypoint shred version
https://github.com/solana-labs/solana/blob/ed51cde37/validator/src/main.rs#L390-L412
This commit increases the min number of bloom items when making gossip
pull requests. Effectively this will break the entrypoint crds table
into 64 shards, one pull-request for each, a larger bloom filter for
each shard, and increases the chances that the response will include
entrypoint's contact-info, which is needed for adopting shred version
and validator start.
2021-05-21 13:59:26 +00:00
13b032b2d4
removes manual trait impl for contact-info ( #17332 )
...
The current implementations use only the id and disregard other fields,
in particular wallclock. This can lead to bugs where an outdated
contact-info shadows or overrides a current one because they compare
equal.
2021-05-19 20:56:10 +00:00
e7073ecab1
adds gossip metrics for number of staked nodes ( #17330 )
2021-05-19 19:25:21 +00:00
0781fe1b4f
Upgrade Rust to 1.52.0 ( #17096 )
...
* Upgrade Rust to 1.52.0
update nightly_version to newly pushed docker image
fix clippy lint errors
1.52 comes with grcov 0.8.0, include this version to script
* upgrade to Rust 1.52.1
* disabling Serum from downstream projects until it is upgraded to Rust 1.52.1
2021-05-19 09:31:47 -05:00
827355a6b1
Create solana-rpc crate and move subscriptions ( #17320 )
...
* Move non_circulating_supply to runtime
* Add solana-rpc crate and move max_slots
* Move subscriptions to solana-rpc
* Single use statements
2021-05-19 00:54:28 -06:00
f7b0184f81
patches flaky test_new_mark_creation_time ( #17288 )
2021-05-18 13:39:35 +00:00
67e6a3106f
rpc: plumb shred_version through RpcContactInfo
2021-05-14 08:36:08 +00:00
27004f1b76
Return error for excluded secondary-index keys ( #17193 )
...
* Add runtime helpers to check secondary indexes for key
* Add custom rpc error
* Check secondary-index key inclusion in rpc
* Clone complete AccountSecondaryIndexes into rpc to avoid bank query
2021-05-13 21:04:21 +00:00
0e646d10bb
prunes received-cache only once per unique owner's key ( #17039 )
2021-05-13 13:50:16 +00:00
0aa7824884
retains one node-instance per pubkey ( #17187 )
...
crds table retains up to 32 node-instance values per each pubkey. This
is so because if there are multiple running instances of the same node,
then we want gossip to propagate node-instance values associated with
both instances, therefore the corresponding label/key includes the
randomly generated token in addition to the pubkey:
https://github.com/solana-labs/solana/blob/9c42a89a4/core/src/crds_value.rs#L448
https://github.com/solana-labs/solana/pull/14037
As a result, the number of such values per pubkey are effectively
unbounded, requiring custom mitigations implemented in:
https://github.com/solana-labs/solana/pull/14467
but still taking redundant extra memory and bandwidth.
This commit instead retains only one node-instance per pubkey by
extending crds values override logic. If a crds value is of type
node-instance, it will always override an existing one with the same key
if it has more recent starting timestamp (not wallclock). As a result,
gossip will always propagate the node-instance with more recent
timestamp. Since the check_duplicate logic will stop the node with older
timestamp, this change should preserve existing functionality.
2021-05-13 13:35:46 +00:00
9c42a89a43
Issue #17008 -- make snapshot archives to hold on to configurable. ( #17158 )
...
* purge_old_snapshot_archives is changed to take an extra argument 'maximum_snapshots_to_retain' to control the max number of latest snapshot archives to retain. Note the oldest snapshot is always retained as before and is not subjected to this new options.
* The validator and ledger-tool executables are modified with a CLI argument --maximum-snapshots-to-retain. And the options are propagated down the call chains. Their corresponding shell scripts were changed accordingly.
* SnapshotConfig is modified to have an extra field for the maximum_snapshots_to_retain
* Unit tests are developed to cover purge_old_snapshot_archives
2021-05-12 10:32:27 -07:00
6e9deaf1bd
Move block-time caching earlier ( #17109 )
...
* Require that blockstore block-time only be recognized slot, instead of root
* Move cache_block_time to after Bank freeze
* Single use statement
* Pass transaction_status_sender by reference
* Remove unnecessary slot-existence check before caching block time altogether
* Move block-time existence check into Blockstore::cache_block_time, Blockstore no longer needed in blockstore_processor helper
2021-05-10 13:14:56 -06:00
f39dda00e0
type AccountSecondaryIndexes = HashSet ( #17108 )
2021-05-10 14:22:48 +00:00
81ad795d46
removes position field in coding-shred-header
...
CodingShredHeader.position is equal to
ShredCommonHeader.index - ShredCommonHeader.fec_set_index
and is so redundant. The extra position field can add bugs if not
consistent with index and fec_set_index.
2021-05-10 13:20:56 +00:00
22c02b917e
reads gossip push messages off crds ordinal index
...
Having an ordinal index on crds values based on insert order allows to
efficiently filter values using a cursor. In particular
CrdsGossipPush::push_messages hash-map can be replaced with a cursor,
saving on the bookkeepings, purging, etc
2021-05-09 22:40:41 +00:00
dfa3e7a61c
indexes crds values by their insert order
2021-05-09 22:40:41 +00:00
d6c076f1b6
getBlockProduction now correctly reports block production
2021-05-07 19:04:51 -07:00
fa86a335b0
implements cursor for gossip crds table queries ( #16952 )
...
VersionedCrdsValue.insert_timestamp is used for fetching crds values
inserted since last query:
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L1197-L1215
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L1274-L1298
So it is crucial that insert_timestamp does not go backward in time when
new values are inserted into the table. However std::time::SystemTime is
not monotonic, or due to workload, lock contention, thread scheduling,
etc, ... new values may be inserted with a stalled timestamp way in the
past. Additionally, reading system time for the above purpose is
inefficient/unnecessary.
This commit adds an ordinal index to crds values indicating their insert
order. Additionally, it implements a new Cursor type for fetching values
inserted since last query.
2021-05-06 14:04:17 +00:00
9ba2c53b85
Add --tower argument to specify where tower files are persisted
2021-05-05 12:20:39 -07:00
f17b80236f
test-validator: Plumb --limit-ledger-size
2021-05-04 08:45:24 +00:00
bc7e741514
Integrate gossip votes into switching threshold ( #16973 )
2021-05-04 00:51:42 -07:00
6318705607
Add keys
2021-05-03 17:18:54 -07:00
b948a18841
Key rotation
2021-05-03 17:18:54 -07:00
b2778f34f5
Rotate keys
2021-05-03 17:18:54 -07:00
7cea2c4466
validates gossip addresses before sending pull-requests
...
IP addresses need to be validated before sending packets to them.
This commit, sends a ping packet to nodes before any pull requests.
Pull requests are then only sent to the nodes which have responded with
the correct hash of their respective ping packet.
2021-05-03 18:21:06 +00:00
2231017b35
uses Mutex instead of RwLock for ping_cache
2021-05-03 18:21:06 +00:00
a698e34744
patches local pending push messages processing ( #16833 )
...
process_push_messages writes local pending push messages to the crds
table, but it discards the return value:
https://github.com/solana-labs/solana/blob/cf779c63c/core/src/crds_gossip.rs#L96-L102
In order to exclude outdated values from the next pull-request, we need
to record the hash of values purged/overridden by the local push
messages, otherwise pull-responses will return outdated values back to
the node:
https://github.com/solana-labs/solana/blob/c1829dd00/core/src/crds_gossip_pull.rs#L447-L452
Additionally, gossip packets arrive and are processed out of order. So,
local pending push messages should be flushed *before* generating bloom
filters for pull-requests, preventing pull-responses returning the same
values back to the node itself. This requires flipping order of
generating pull and push messages:
https://github.com/solana-labs/solana/blob/cf779c63c/core/src/cluster_info.rs#L1757-L1762
Both above bugs cause redundant traffic and bandwidth waste in gossip
pull-responses.
2021-05-03 16:00:17 +00:00
541aa5ad85
tests: lamports -> lamports() ( #16982 )
2021-05-03 10:45:54 -05:00
8e561354d5
Improve readability of vote lockout processing ( #16987 )
...
* Improve readability of vote lockout processing
* clippy
* simplify comment
* feedback
2021-05-02 08:36:06 +00:00
5981399612
Distinguish max replayed and max observed vote ( #16936 )
2021-04-29 14:43:28 -07:00
542d88929f
Add getBlockProduction RPC method
2021-04-28 20:02:54 -07:00
b5d30846d6
Retry latest vote if expired ( #16735 )
2021-04-28 11:46:16 -07:00
25054bfd35
retains peer's contact-info when making pull requests ( #16715 )
...
ClusterInfo::new_pull_requests has to lookup contact-infos:
https://github.com/solana-labs/solana/blob/a1ef2bd74/core/src/cluster_info.rs#L1663-L1673
when it was already available when making pull requests:
https://github.com/solana-labs/solana/blob/a1ef2bd74/core/src/crds_gossip_pull.rs#L232
2021-04-28 13:19:12 +00:00
1ac2a8cfa5
removes delayed crds inserts when upserting gossip table ( #16806 )
...
It is crucial that VersionedCrdsValue::insert_timestamp does not go
backward in time:
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/crds.rs#L67-L79
Otherwise methods such as get_votes and get_epoch_slots_since will
break, which will break their downstream flow, including vote-listener
and optimistic confirmation:
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L1197-L1215
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L1274-L1298
For that, Crds::new_versioned is intended to be called "atomically" with
Crds::insert_verioned (as the comment already says so):
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/crds.rs#L126-L129
However, currently this is violated in the code. For example,
filter_pull_responses creates VersionedCrdsValues (with the current
timestamp), then acquires an exclusive lock on gossip, then
process_pull_responses writes those values to the crds table:
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L2375-L2392
Depending on the workload and lock contention, the insert_timestamps may
well be in the past when these values finally are inserted into gossip.
To avoid such scenarios, this commit:
* removes Crds::new_versioned and Crd::insert_versioned.
* makes VersionedCrdsValue constructor private, only invoked in
Crds::insert, so that insert_timestamp is populated right before
insert.
This will improve insert_timestamp monotonicity as long as Crds::insert
is not called with a stalled timestamp. Following commits may further
improve this by calling timestamp() inside Crds::insert, and/or
switching to std::time::Instant which guarantees monotonicity.
2021-04-28 11:56:13 +00:00
b17d5eeaee
moves cluster-info metrics to a separate module ( #16883 )
2021-04-28 02:04:49 +00:00
b468ead1b1
uses current timestamp when flushing local pending push queue ( #16808 )
...
local_message_pending_push_queue is recording timestamps at the time the
value is created, and uses that when the pending values are flushed:
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L321
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/crds_gossip.rs#L96-L102
which is then used as the insert_timestamp when inserting values in the
crds table:
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/crds_gossip_push.rs#L183
The flushing may happen 100ms after the values are created (or even
later if there is a lock contention). This will cause non-monotone
insert_timestamps in the crds table (where time goes backward),
hindering the usability of insert_timestamps for other computations.
For example both ClusterInfo::get_votes and get_epoch_slots_since rely
on monotone insert_timestamps when values are inserted into the table:
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L1197-L1215
https://github.com/solana-labs/solana/blob/ec37a843a/core/src/cluster_info.rs#L1274-L1298
This commit removes timestamps from local_message_pending_push_queue and
uses current timestamp when flushing the queue.
2021-04-28 00:15:11 +00:00
bc31378797
Trim extra shred bytes in blockstore ( #16602 )
...
Strip the zero-padding off of data shreds before insertion into blockstore
Co-authored-by: Stephen Akridge <sakridge@gmail.com>
Co-authored-by: Nathan Hawkins <utsl@utsl.org>
2021-04-27 17:40:41 -05:00
3b8d6b59fb
records hash of values purged by expired pull-responses ( #16800 )
...
process_pull_responses should record hash of values purged by expired
responses (as well as unexpired ones):
https://github.com/solana-labs/solana/blob/c1829dd00/core/src/crds_gossip_pull.rs#L385-L387
otherwise, these values are not excluded from following pull-requests
(from likely different nodes):
https://github.com/solana-labs/solana/blob/c1829dd00/core/src/crds_gossip_pull.rs#L447-L452
and would waste bandwidth should they be included in subsequent
pull-responses.
2021-04-27 12:06:49 +00:00
0f3ac51cf1
limits to data_header.size when combining shreds' payloads ( #16708 )
...
Shredder::deshred is ignoring data_header.size when combining shreds' payloads:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/shred.rs#L940-L961
Also adding more sanity checks on the alignment of data shreds indices.
2021-04-27 12:04:44 +00:00
59fc33635a
Add getVoteAccounts RPC method parameter to restrict results to a single vote account
2021-04-27 04:27:15 +00:00
9706512115
removes old runtime feature gates in gossip and turbine ( #16633 )
2021-04-26 17:12:02 +00:00
ca14c18998
owner -> owner() ( #16782 )
2021-04-23 22:49:47 +00:00
63436cc2bf
Disable flaky test_poh_service ( #16772 )
2021-04-23 12:14:11 -05:00
2c82f2154d
retains crds values if the origin is still active ( #16576 )
...
Local timestamps are updated for records associated with a pubkey if the
origin is still active:
https://github.com/solana-labs/solana/blob/c8ed14c64/core/src/crds.rs#L301-L311
However this is done inconsistently on some gossip paths (pull requests
and pull responses) but not all (e.g. push messages). Additionally
update_record_timestamp is inefficient since there can be ~800 values
associated with each pubkey.
This commit updates records timestamps only on contact-infos; and,
instead utilizes origin's timestamp when purging old values.
2021-04-23 15:14:49 +00:00
03194145c0
removes first_coding_index from erasure recovery code ( #16646 )
...
first_coding_index is the same as the set_index and is so redundant:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/blockstore_meta.rs#L49-L60
2021-04-23 12:00:37 +00:00
75b8434b76
Add TPU client for sending txs to the current leader tpu port ( #16736 )
...
* Add TPU client for sending txs to the current leader tpu port
* Update tpu_client.rs
2021-04-23 09:35:12 +08:00
636b5987af
Update getLeaderSchedule options ( #16749 )
2021-04-22 19:27:30 +00:00
6004c0abf5
getLeaderSchedule now supports filtered results based on validator identity
2021-04-21 17:59:26 -07:00
91b6888e15
verify_pubkey() now takes a ref
2021-04-21 14:43:49 -07:00
4c94f8933f
Ingest votes from gossip into fork choice ( #16560 )
2021-04-21 14:40:35 -07:00
a1ef2bd74d
Ignore flaky test_pull_request_time_pruning
2021-04-21 12:07:36 -07:00
37b8587d4e
expands number of erasure coding shreds in the last batch in slots ( #16484 )
...
Number of parity coding shreds is always less than the number of data
shreds in FEC blocks:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L719
Data shreds are batched in chunks of 32 shreds each:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L714
However the very last batch of data shreds in a slot can be small, in
which case the loss rate can be exacerbated.
This commit expands the number of coding shreds in the last FEC block in
slots to: 64 - number of data shreds; so that FEC blocks are always 64
data and parity coding shreds each.
As a consequence of this, the last FEC block has more parity coding
shreds than data shreds. So for some shred indices we will have a coding
shred but no data shreds. This should not cause any kind of overlapping
FEC blocks as in:
https://github.com/solana-labs/solana/pull/10095
since this is done only for the very last batch in a slot, and the next
slot will reset the shred index.
2021-04-21 12:47:50 +00:00
0924c2d070
Add port and gossip options to solana-test-validator ( #16696 )
2021-04-21 02:40:52 +00:00
34addee882
getVoteAccounts: Limit the length of the `epoch_credits` array
2021-04-20 14:42:28 -07:00
8e69dd42c1
Add non-default repair nonce values ( #16512 )
...
* Track outstanding nonces in repair
* Rework outstanding requests to use lru cache and randomize nonces
Co-authored-by: Carl <carl@solana.com>
2021-04-20 09:37:33 -07:00
bc90e04e64
uses current local timestamp when recording purged values
...
CrdsGossipPull.purged_values is meant to record recently purged values
so that they are excluded from imminent pull requests, until the entire
cluster have synced to the updated value:
https://github.com/solana-labs/solana/blob/c826cddbb/core/src/crds_gossip_pull.rs#L449-L454
However, VersionedCrdsValue.local_timestamp represents the local time
when the value was last updated, and given that crds values may have
different timeouts based on stake, it does not necessarily represent how
recently the value was purged:
https://github.com/solana-labs/solana/blob/c826cddbb/core/src/crds.rs#L75-L76
As such, recording current local timestamp when purging values is more
appropriate. Additionally, purge_purged assumes that the purge_values is
sorted in timestamps when draining the old ones; which is not true if
those timestamps are VersionedCrdsValue.local_timestamp:
https://github.com/solana-labs/solana/blob/c826cddbb/core/src/crds_gossip_pull.rs#L563-L571
2021-04-20 11:21:00 +00:00
a7e65c0034
RPC: use finalized as default pubsub commitment level ( #16659 )
...
* RPC: use finalized as default pubsub commitment level
* update docs
* Fix tests
2021-04-20 08:19:54 +00:00
c8b474cd0b
Send votes to next leader's TPU instead of our TPU
2021-04-20 00:38:21 -07:00
4aa753ff01
rename threads: 15 char limit ( #16625 )
2021-04-19 12:16:58 -05:00
b06e93fe5b
Increase test timeout
2021-04-18 20:55:02 -07:00
a911ae00ba
clippy
2021-04-18 20:55:02 -07:00
e405747409
Revert "Add limit and shrink policy for recycler ( #15320 )"
...
This reverts commit c2e8814dce
2021-04-18 19:29:24 +00:00
6907a2366e
Remove unnecessary clone
2021-04-17 10:23:13 -07:00
bb24318ef0
Document shreds ( #16514 )
...
No functionality changes from this commit
2021-04-16 14:04:46 -05:00
974e6dd2c1
Deprecate "confirmed" RpcClient methods ( #16520 )
...
* Remove obsolete client methods
* Deprecate GetConfirmed client methods
* Rename Confirmed config structs, with appropriate deprecation
* Fixup client apps
* Map RpcRequest to deprecated when targeting older nodes
2021-04-15 17:00:14 -06:00
7dfb51c0b4
Cli: move airdrop to rpc requests ( #16557 )
...
* Add recent_blockhash to requestAirdrop
* Move tx confirmation to separate method
* Add RpcClient airdrop methods
* Request cli airdrop via RpcClient
* Pass optional faucet_addr into TestValidator and fix tests
* Update client/src/rpc_client.rs
Co-authored-by: Michael Vines <mvines@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-04-15 06:25:23 +00:00
d92721aab9
uses timeouts based on stake for filtering pull responses ( #16549 )
...
filter_pull_responses is using default timeout when discarding pull
responses (except for ContactInfo):
https://github.com/solana-labs/solana/blob/f804ce63c/core/src/crds_gossip_pull.rs#L349-L350
But purging code uses timeouts based on stake:
https://github.com/solana-labs/solana/blob/f804ce63c/core/src/cluster_info.rs#L1867-L1870
So the crds value will not be purged from the sender's table and will be
sent again over the next pull request.
2021-04-14 20:18:00 +00:00
f35a6a8be0
prioritizes contact-infos in pull responses ( #16541 )
...
Expired crds values where the contact-info does not exist are wasted:
https://github.com/solana-labs/solana/blob/f804ce63c/core/src/crds_gossip_pull.rs#L353-L378
and then are sent again over the next pull-request.
Also, the stake of the first response (which can be anything) is used to
weight all pull-responses to a node, while the rest of responses can
have different stake.
https://github.com/solana-labs/solana/blob/f804ce63c/core/src/cluster_info.rs#L2231
2021-04-14 18:45:20 +00:00
f0c150cfb9
Fix channel panic in tests ( #16503 )
...
* Fix channel panic
* Add exit signal to PohRecorder because Crossbeam doesnt drop objects inside dropped channel
2021-04-14 12:07:21 -05:00
f641429056
Remove blake3 from bpf program dependencies ( #16506 )
2021-04-13 10:06:05 +00:00
37afa00ffb
Rpc: deprecate getConfirmed endpoints ( #16502 )
...
* Deprecate getConfirmed methods in rpc
* Add new methods to docs
* Move deprecated rpc methods to separate docs section
* Add note to docs about removal timing
2021-04-13 01:50:15 -06:00
85eb37fab0
Merge pull request from GHSA-8v47-8c53-wwrc
...
* Track transaction check time separately from account loads
* banking packet process metrics
* Remove signature clone in status cache lookup
* Reduce allocations when converting packets to transactions
* Add blake3 hash of transaction messages in status cache
* Bug fixes
* fix tests and run fmt
* Address feedback
* fix simd tx entry verification
* Fix rebase
* Feedback
* clean up
* Add tests
* Remove feature switch and fall back to signature check
* Bump programs/bpf Cargo.lock
* clippy
* nudge benches
* Bump `BankSlotDelta` frozen ABI hash`
* Add blake3 to sdk/programs/Cargo.lock
* nudge bpf tests
* short circuit status cache checks
Co-authored-by: Trent Nelson <trent@solana.com>
2021-04-13 00:28:08 -06:00
70f3f7e679
Move obsolete rpc endpoints to separate api for removal ( #16500 )
...
* Move obsolete rpc methods to separate api for removal
* Remove obsolete method from docs
* Fix test using obs method
2021-04-12 20:33:40 -06:00
2229b70c4e
Add authorized-voter add/remove-all commands
2021-04-12 15:55:28 -07:00
17a173ebb5
Clean up build warning
2021-04-12 15:55:28 -07:00
dc7030ffaa
Allow fork choice to support multiple versions of a slot ( #16266 )
2021-04-12 01:00:59 -07:00
99b3aab703
Track gossip vote updates per hash for replay stage ( #16421 )
...
* Track gossip vote updates per hash for replay stage
2021-04-10 17:34:45 -07:00
54a04bac3d
Apple M1 compatibility ( #16346 )
...
Co-authored-by: Christian Drappi <christiandrappi@Christians-MacBook-Pro.local>
2021-04-09 17:21:01 -07:00
8bc0bdd40b
Fill in not-yet-finalized block-time if possible ( #16460 )
2021-04-09 20:25:47 +00:00
22a18a68e3
stops consuming pinned vectors with a recycler ( #16441 )
...
If the vector is pinned and has a recycler, From<PinnedVec>
implementation of Vec should clone (instead of consuming) the underlying
vector so that the next allocation of a PinnedVec will recycle an
already pinned one.
2021-04-09 16:55:24 +00:00
b08cff9e77
Simplify some pattern-matches ( #16402 )
...
When those match an exact combinator on Option / Result.
Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust ).
2021-04-08 12:40:37 -06:00
bb9d2fd07a
Cli: use get_inflation_rewards and limit epochs queried ( #16408 )
...
* Fix block-with-limit when not finalized blocks found
* Enable confirmed commitment in getInflationReward
* Use get_inflation_rewards in cli
* Line up rewards output
* Add range validator
* Change cli epoch arg -> num epochs
* Add solana inflation rewards subcommand
* Consolidate epoch rewards meta
2021-04-08 10:57:33 -06:00
e501fa5f0b
Rpc: introduce get_inflation_reward rpc call ( #16278 )
...
* feat: introduce get_inflation_reward rpc call
* fix: style suggestions
* fix: more style changes and match how other rpc functions are defined
* feat: get reward for a single epoch
* feat: default to the most recent epoch
* fix: don't factor out get_confirmed_block
* style: introduce from impl for RpcEncodingConfigWrapper
* style: bring commitment into variable
* feat: support multiple pubkeys for get_inflation_reward
* feat: add get_inflation_reward to rpc client
* feat: return rewards in order
* fix: rename pubkeys to addresses
* docs: introduce jsonrpc docs for get_inflation_reward
* style: early return in map (not sure which is more idiomatic)
* fix: call the rpc client function args addresses as well
* fix: style
* fix: filter out only addresses we care about
* style: make this more idiomatic
* fix: change rpc client epoch to optional and include some docs edits
* feat: filter out rent rewards in get_inflation_reward
* feat: add option epoch config param to get_inflation_reward
* feat: rpc client get_inflation_reward takes epoch instead of config and some filter staking and voting rewards
2021-04-06 18:10:53 -07:00
1219842a96
No wallclock throttle tests ( #16396 )
2021-04-05 19:40:16 -07:00
b71875df61
cluster-info: Get rid of some integer math while we're here
2021-04-06 00:09:37 +00:00
b6b08706b9
cluster-info: Don't subtract non-shred spies from node count
2021-04-06 00:09:37 +00:00
7a2a39093d
validator: Use a const for wait for supermajority threshold
2021-04-05 17:29:37 -06:00
43feef7362
Adjust tokio version to just "1"
2021-04-05 17:13:20 +00:00
b242f82696
Reduce test-validator ledger size
2021-04-05 08:37:29 -07:00
701fc93343
patches bug in banking stage where buffered packets are never retained ( #16276 )
...
banking_stage::handle_forwarding is retaining buffered packets with
empty index, so nothing is held:
https://github.com/solana-labs/solana/blob/6f3926b64/core/src/banking_stage.rs#L520
2021-04-05 12:46:21 +00:00
3429785d9b
Wait for 90 percent of stake before starting ( #16340 )
2021-04-03 14:21:20 -07:00
4e5ef6bce2
Add cluster state verifier logging ( #16330 )
...
* Add cluster state verifier logging
* Add duplicate-slots iterator to ledger tool
2021-04-02 21:48:44 -07:00
da27acabcc
Rpc: enable getConfirmedSignaturesForAddress2 to return confirmed (not yet finalized) data ( #16281 )
...
* Update blockstore method to allow return of unfinalized signature
* Support confirmed sigs in getConfirmedSignaturesForAddress2
* Add deprecated comments
* Update docs
* Enable confirmed transaction-history in cli
* Return real confirmation_status; fill in not-yet-finalized block time if possible
2021-04-01 04:35:57 +00:00
18bd47dbe1
Rpc: fix getConfirmedTransaction slot ( #16288 )
...
* Fix transaction blockstore apis
* Update blockstore apis in rpc
2021-03-31 21:04:00 -06:00
3f63ed9a72
removes OrderedIterator and transaction batch iteration order ( #16153 )
...
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817
However load_and_execute_transactions is iterating over
lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.
If iteratorion_order is [1, 2, 0], and i is 0, then:
lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).
This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.
2021-03-31 23:59:19 +00:00
54c68ea83f
Drop write lock on sysvars ( #15497 )
...
* Drop write lock on sysvars
* adds env var for demoting sysvar write lock demotion
* moves demote logic to is_writable
* feature gates sysvar write lock demotion
* adds builtins to write lock demotion
* adds system program id to builtins
* adds Feature111...
* adds an abi-freeze test
* mvines set of builtin program keys
Co-authored-by: Michael Vines <mvines@gmail.com>
* update tests
* adds bpf loader keys
* Add test sysvar
* Plumb demote_sysvar to is_writable
* more plumbing of demote_sysvar_write_locks to is_writable
* patches test_program_bpf_instruction_introspection
* hard codes demote_sysvar_write_locks to false for serialization/encoding methods
* Revert "hard codes demote_sysvar_write_locks to false for serialization/encoding methods"
This reverts commit ae3e2d2e777437bddd753933097a210dcbc1b1fc.
* change the hardcoded ones to demote_sysvar_write_locks=true
* Use data_as_mut_slice
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-03-30 10:05:09 -07:00
414c7070cb
poll checking for new record in poh service after every batch of hashes instead of busy waiting ( #16167 )
...
* poll waiting in poh service after every batch of hashes
* clippy
2021-03-30 10:34:21 -05:00
5eff23db0c
TransactionRecorder uses unique channel so we can use Recv instead of RecvTimeout ( #16195 )
...
* time
* new channel each call
* new channel every time
2021-03-30 00:51:35 -05:00
3977ed5c82
Future-aware enum name
2021-03-29 14:58:05 -06:00
60ed8e2892
Rpc: enable getConfirmedBlocks and getConfirmedBlocksWithLimit to return confirmed (not yet finalized) data ( #16161 )
...
* Add commitment config capabilities
* Use rpc limit if no end_slot provided
* Limit to actually finalized blocks
* Support confirmed blocks in getConfirmedBlocks and getConfirmedBlocksWithLimit
* Update docs
* Add client plumbing
* Rename config enum
2021-03-29 12:41:31 -06:00
60b4771fc6
Only print skipped leader slot message when the node is actually leader ( #16156 )
...
Also, check vote signature after the vote is signed
2021-03-26 17:45:53 -07:00
433f1ead1c
Rpc: enable getConfirmedBlock and getConfirmedTransaction to return confirmed (not yet finalized) data ( #16142 )
...
* Add Blockstore block and tx apis that allow unrooted responses
* Add TransactionStatusMessage, and send on bank freeze; also refactor TransactionStatusSender
* Track highest slot with tx-status writes complete
* Rename and unpub fn
* Add commitment to GetConfirmed input configs
* Support confirmed blocks in getConfirmedBlock
* Support confirmed txs in getConfirmedTransaction
* Update sigs-for-addr2 comment
* Enable confirmed block in cli
* Enable confirmed transaction in cli
* Review comments
* Rename blockstore method
2021-03-26 16:47:35 -06:00
4f4cffbd03
Throttle PoH ticks by cumulative slot time ( #16139 )
...
* Throttle PoH ticks by cumulative slot time
* respond to pr feedback
* saturating sub
* updated comment
2021-03-26 18:54:16 +00:00
06ac0fe9a3
increase timeout in TransactionRecorder.record ( #16133 )
2021-03-25 21:31:07 -05:00
b99ae8f334
Skip leader slots until a vote lands ( #15607 )
2021-03-25 18:54:51 -07:00
b041b55028
makes test_pull_request_time_pruning smaller ( #16128 )
2021-03-25 22:44:43 +00:00
9b94741290
Fix test_replay_commitment_cache ( #16131 )
2021-03-25 21:16:39 +00:00
e817a6db00
Add timeout for local cluster partition tests ( #16123 )
...
* Add timeout for local cluster partition tests
* fix optimistic conf test logs
* Bump instruction count assertions
2021-03-25 13:27:07 -06:00
52703badfa
Setup ReplayStage confirmation scaffolding for duplicate slots ( #9698 )
2021-03-24 23:41:52 -07:00
a8ef29df27
Support getBlockTime for unfinalized blocks ( #16103 )
2021-03-24 20:52:08 -06:00
96ccc40f0a
Set ticks_per_slot higher for banking stage tests ( #16094 )
...
Tests are timing out because the bank hit the MaxTickHeight and
will not process the transactions.
2021-03-24 14:05:43 -07:00
f68860a643
poh record metrics ( #16092 )
2021-03-24 14:48:32 -05:00
a6c23648cb
limits CrdsGossipPull::pull_request_time size ( #15793 )
...
There is no pruning logic on CrdsGossipPull::pull_request_time
https://github.com/solana-labs/solana/blob/79ac1997d/core/src/crds_gossip_pull.rs#L172-L174
potentially allowing this to take too much memory.
Additionally, CrdsGossipPush::last_pushed_to is pruning recent push
timestamps:
https://github.com/solana-labs/solana/blob/79ac1997d/core/src/crds_gossip_push.rs#L275-L279
instead of the older ones.
Co-authored-by: Nathan Hawkins <utsl@utsl.org>
2021-03-24 18:33:56 +00:00
570fd3f810
makes turbine peer computation consistent between broadcast and retransmit ( #14910 )
...
get_broadcast_peers is using tvu_peers:
https://github.com/solana-labs/solana/blob/84e52b606/core/src/broadcast_stage.rs#L362-L370
which is potentially inconsistent with retransmit_peers:
https://github.com/solana-labs/solana/blob/84e52b606/core/src/cluster_info.rs#L1332-L1345
Also, the leader does not include its own contact-info when broadcasting
shreds:
https://github.com/solana-labs/solana/blob/84e52b606/core/src/cluster_info.rs#L1324
but on the retransmit side, slot leader is removed only _after_ neighbors and
children are computed:
https://github.com/solana-labs/solana/blob/84e52b606/core/src/retransmit_stage.rs#L383-L384
So the turbine broadcast tree is different between the two stages.
This commit:
* Removes retransmit_peers. Broadcast and retransmit stages will use tvu_peers
consistently.
* Retransmit stage removes slot leader _before_ computing children and
neighbors.
2021-03-24 13:34:48 +00:00
e7fd7d46cf
rpc: add getSlotLeaders method ( #16057 )
2021-03-23 17:48:54 +00:00
4f82b897bc
buffers data shreds to make larger erasure coded sets ( #15849 )
...
Broadcast stage batches up to 8 entries:
https://github.com/solana-labs/solana/blob/79280b304/core/src/broadcast_stage/broadcast_utils.rs#L26-L29
which will be serialized into some number of shreds and chunked into FEC
sets of at most 32 shreds each:
https://github.com/solana-labs/solana/blob/79280b304/ledger/src/shred.rs#L576-L597
So depending on the size of entries, FEC sets can be small, which may
aggravate loss rate.
For example 16 FEC sets of 2:2 data/code shreds each have higher loss
rate than one 32:32 set.
This commit broadcasts data shreds immediately, but also buffers them
until it has a batch of 32 data shreds, at which point 32 coding shreds
are generated and broadcasted.
2021-03-23 14:52:38 +00:00
57ba86c821
eliminate lock on record ( #15929 )
...
* eliminate lock on record
* use same error as MaxHeightReached
* clippy
* review feedback
* refactor should_tick code
* pr feedback
2021-03-23 09:10:04 -05:00
2fc609a294
add metric for ticks from poh_recorder.record ( #16047 )
2021-03-22 15:35:06 -05:00
2ec24d438f
Make getStakeActivation response consistent for undelegated accounts ( #16038 )
2021-03-19 14:54:56 -06:00
ddc758439e
metrics for poh_recorder.record ( #15998 )
2021-03-19 09:48:55 -05:00
02b81dd05d
Update to reqwest 0.11.2
2021-03-18 11:59:41 -07:00
aa54c468ea
rpc: Add config options limiting getConfirmedBlock response data ( #15970 )
...
* Add new confirmed block struct
* Add RpcConfirmedBlockConfig options
* Configure block response based on new options
* Add client api, use in cli fetch_epoch_rewards
* Update docs
* Apply review suggestions
2021-03-18 17:58:20 +00:00
04c99cf7ea
Add --slots-per-epoch argument
2021-03-17 22:56:41 +00:00
f548a04fae
Allow unbounded wallclock processing time in tests ( #15961 )
2021-03-17 15:48:50 -07:00
59c19d9fbf
Notice the user when the --mint, --bpf-program, or --clone arguments are ignored
2021-03-17 20:04:53 +00:00
8a9b51952e
Ignore flaky test_banking_stage_entries_only and test_banking_stage_entryfication
2021-03-17 11:28:56 -07:00
40997d0aef
add metrics for tick producer and poh_recorder ( #15931 )
2021-03-17 10:38:26 -05:00
5460fb10a2
drop poh lock after record ( #15930 )
2021-03-17 10:37:20 -05:00
a2eb655322
=1.7.0
2021-03-16 07:51:07 +00:00
efee8b62d7
a few missed set_data calls ( #15846 )
...
* a few missed set_data calls
* another set data call
2021-03-15 21:57:23 -05:00
c09ea2c314
More AccountSharedData construction ( #15844 )
...
* one more AccountSharedData construction
* one more construct
2021-03-15 19:27:17 -05:00
c1ba265dd9
Wallclock BankingStage Throttle ( #15731 )
2021-03-15 17:11:15 -07:00
0c9ca5522c
Bump version to v1.7.0
2021-03-13 09:01:21 +00:00
5b2da19c93
Rpc: support extended config for getConfirmedBlock ( #15827 )
...
* Add rpc confirmed-block config wrapper to support struct of extended config
* Update docs
* Make config wrapper generic and use in getConfirmedTransaction as well
* Update/clean confirmed-tx docs
2021-03-12 22:19:45 +00:00
f2865dfd63
requires stakes for propagating crds values through gossip ( #15561 )
2021-03-12 15:50:14 +00:00
918d04e3f0
Add more slot update notifications ( #15734 )
...
* Add more slot update notifications
* fix merge
* Address feedback and add integration test
* switch to datapoint
* remove unused shred method
* fix clippy
* new thread for rpc completed slots
* remove extra constant
* fixes
* rely on channel closing
* fix check
2021-03-12 21:44:06 +08:00
4bbeb9c033
Remove old feature: simple_capitalization ( #15763 )
...
* Remove old feature: simple_capitalization
* Fix another failing test in core
* Finish up test cleanup
* Further clean up a bit
2021-03-12 11:12:40 +09:00
952c3bcbb7
AccountSharedData construction ( #15790 )
2021-03-11 18:09:04 -06:00
24d18b3cf2
docs: add docs links for crates published to crates.io
2021-03-11 08:38:18 +00:00
1135ffd595
mut data refs as slice ( #15782 )
2021-03-10 15:28:03 -06:00
56923c91bf
limits number of unique pubkeys in the crds table ( #15539 )
2021-03-10 20:46:05 +00:00
52e54e1100
account.data -> data() ( #15778 )
2021-03-09 22:31:33 +00:00
8a3135d17b
Account->AccountSharedData ( #15691 )
2021-03-09 15:06:07 -06:00
2bee9435f3
Add tracer key for tracing transaction path through the network ( #15732 )
2021-03-08 19:31:00 -08:00
331c45decf
Report datapoint on number of retransmit shreds ( #15694 )
2021-03-08 17:54:53 -08:00
d09112fa6d
PoH batch size calibration ( #15717 )
2021-03-05 16:01:21 -08:00
4a3ab77baf
Remove unused id field
2021-03-05 19:07:59 +00:00
66b781eec3
Add 'unknown' health check state
2021-03-05 17:46:50 +00:00
7e65289729
Convert blockstore TransactionStatus column family to protobufs ( #15733 )
...
* Prevent panic if TransactionStatus can't be deserialized
* Convert Blockstore TransactionStatus column to protobuf
* Add compatability test
2021-03-05 09:05:35 -07:00
bd13262b42
Add validator startup process reporting before RPC is available
2021-03-05 08:03:36 -08:00
24ab84936e
Break up RPC API into three categories: minimal, full and admin
2021-03-04 16:39:44 -08:00
34bebb7d09
report execution details in replay time ( #15693 )
2021-03-04 11:38:12 -06:00
be35c1c1b7
add execute detail timings ( #15638 )
2021-03-03 17:07:45 -06:00
658951e680
sends only the latest vote of each validator to the banking stage ( #15629 )
2021-03-03 19:07:16 +00:00
aacb28c453
Only report metrics every second ( #15652 )
2021-03-03 10:58:47 -08:00
830be855dc
Forward and hold packets ( #15634 )
2021-03-03 10:23:05 -08:00
19ac79b5cc
Deprecate UiTokenAmount::ui_amount ( #15616 )
...
* Add TokenAmount::ui_amount_string
* Fixup solana-tokens
* Update docs
2021-03-02 22:51:41 -07:00
a4f0033bd7
Remove ValidatorConfig derive Clone, and fix local-cluster tests ( #15647 )
...
* Remove ValidatorConfig derive Clone
* Add local-cluster ValidatorConfig helpers
* Fix benches
2021-03-03 04:21:30 +00:00
0bd0084b0d
adds more metrics for tx counts and batch sizes ( #15642 )
2021-03-03 01:28:15 +00:00
416ea38028
adds metrics for the size and number of batches in bank_send_loop ( #15627 )
2021-03-02 15:44:35 +00:00
2463cc1e6a
Fix typos ( #15610 )
2021-03-02 06:36:49 -08:00
640e36287e
Move ValidatorExit into ValidatorConfig, making it accessible from the solana-validator crate
2021-03-01 16:49:56 -08:00
f1223fb783
Lower blockstore processor error severity ( #15578 )
2021-03-01 14:57:37 -08:00
ef63ffce7d
Bump ed25519-dalek to 1.0.1
2021-03-01 21:10:46 +00:00
ae96ba3459
Plumb slot update pubsub notifications ( #15488 )
2021-02-28 23:29:11 -08:00
f7a049f87f
coalesces vote packets into one Packets ( #15566 )
2021-02-26 23:23:08 +00:00
05409e51ce
Increase tpu coalescing and add parameter ( #15536 )
...
Should create larger entries on average
2021-02-26 09:15:45 -08:00
5a9896706c
indexes epoch slots in crds table ( #15459 )
...
ClusterInfo::get_epoch_slots_since scans the entire crds table to obtain
epoch-slots inserted since a timestamp:
https://github.com/solana-labs/solana/blob/013daa8f4/core/src/cluster_info.rs#L1245-L1262
The alternative is to index epoch-slots in crds table ordered by their
insert timestamp.
2021-02-26 14:12:04 +00:00
1ad2c9f741
Revert "Make UiTokenAmount::ui_amount a String ( #15447 )" ( #15542 )
...
This reverts commit d14374bc9f
2021-02-25 21:53:40 +00:00
5b54aed1c0
Speed up getLeaderSchedule
2021-02-24 11:17:25 -08:00
61ed980ac0
Fix received notifications for gossip signature subscriptions ( #15506 )
2021-02-24 16:59:22 +08:00
c2e8814dce
Add limit and shrink policy for recycler ( #15320 )
2021-02-24 00:15:58 -08:00
52f2d425e5
Count if optimistically confirmed slot is already rooted ( #15492 )
2021-02-23 22:03:22 +00:00
1b59b163dd
Add max retransmit and shred insert slot ( #15475 )
2021-02-23 13:06:33 -08:00
4b0114b991
Limit the number of getProgramAccounts filters
2021-02-23 18:43:22 +00:00
d14374bc9f
Make UiTokenAmount::ui_amount a String ( #15447 )
...
* Make UiTokenAmount::ui_amount a String
* Fixup solana-tokens
* Ignore spl downstream-project
2021-02-22 13:05:45 -07:00
5ccaa6336a
Print original error from accounts dir remove ( #15458 )
2021-02-22 21:24:09 +09:00
013daa8f47
RPC: Improve snapshot path sanitization
2021-02-20 13:06:07 -08:00
5df36aec7d
Pacify clippy
2021-02-19 20:08:41 -08:00
fd3b71a2c6
cargo fmt
2021-02-19 20:08:41 -08:00
aa3aac766f
adds metrics for inbound/outbound gossip packets counts ( #15407 )
2021-02-19 22:49:35 +00:00
15bbe6436d
Bump SPL token version to v3.1.0 ( #15429 )
...
* Bump SPL token version to v3.1.0
* Cargo.lock
2021-02-19 09:15:41 +00:00
170cb792eb
Return blockstore error if previous_blockhash cannot be determined ( #15382 )
...
* Return blockstore error if previous_blockhash cannot be determined
* Add require_previous_blockshash flag
2021-02-18 01:04:52 +00:00
7f7370c306
Re-allow clippy::integer_arithmetic at crate-level
2021-02-17 13:55:08 -07:00
9ba69a7381
Adapt to fs_extra 1.2.0
2021-02-17 12:44:58 -08:00
b24cb9840e
Speedup ledger cleanup test ( #15304 )
...
Just clone to produce shreds and use a separate insert thread.
2021-02-17 08:59:25 -08:00
98e3e570d2
Add --force arg for bigtable upload
2021-02-16 17:24:07 -08:00
ba02452d75
add validator flag no-accounts-db-index-hashing ( #15350 )
...
* add validator flag no_accounts_db_index_hashing
* add validator flag no_accounts_db_index_hashing
2021-02-16 21:13:48 +00:00
f79c9d4094
adds an upper bound on cluster-slots size ( #15300 )
...
https://github.com/solana-labs/solana/issues/14366#issuecomment-769096305
2021-02-16 21:12:13 +00:00
076c20f1ca
checks that prune-messages have the same inner/outer pubkey ( #15352 )
2021-02-16 21:06:18 +00:00
b3256aa722
Bump rand_core to 0.6.2
...
https://rustsec.org/advisories/RUSTSEC-2021-0023
2021-02-15 22:51:35 +00:00
5b8f046c67
More configurable rocksdb compaction ( #15213 )
...
rocksdb compaction can cause long stalls, so
make it more configurable to try and reduce those stalls
and also to coordinate between multiple nodes to not induce
stall at the same time.
2021-02-14 10:16:30 -08:00
629dcd0f39
Cleanup buffered packets ( #15210 )
2021-02-12 03:27:37 -08:00
990bb426a9
Fix flaky test test_concurrent_snapshot_packaging ( #15252 )
2021-02-11 16:03:51 -08:00
4013f91dbe
RPC: add caching to getLargestAccounts ( #15154 )
...
* introduce get largest accounts cache
* remove cache size and change hash key
* remove eq and hash derivation from commitment config
* add slot to the cache
2021-02-11 11:32:46 -08:00
948819dfa8
Use spl-token-mint secondary index for relevant getProgramAccounts requests ( #15219 )
2021-02-09 15:49:13 -07:00
da6753b8c0
Warp timestamp and extend max-allowable-drift for accommodate slow blocks ( #15204 )
...
* Remove timestamp_correction feature gating
* Remove timestamp_bounding feature gating
* Remove unused deprecated ledger code
* Remove unused deprecated unbounded-timestamp code
* Enable independent adjustment of fast/slow timestamp bounding
* Update timestamp bounds to 25% fast, 80% slow; warp timestamp
* Update bank hash test
* Add PR number to feature
Co-authored-by: Michael Vines <mvines@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-09 15:49:00 -07:00
2758588ddd
uses btree-map instead of hash-map for cluster-slots ( #15194 )
...
retain traverses all values in the hashmap which is slow:
https://github.com/solana-labs/solana/blob/88f22c360/core/src/cluster_slots.rs#L45
btree-map instead allows more efficient prunning there.
In addition there is potential race condition here:
https://github.com/solana-labs/solana/blob/88f22c360/core/src/cluster_slots.rs#L68-L74
If another thread inserts a value at the same slot key between the read
and write lock, current thread will discard the inserted value.
2021-02-09 22:04:41 +00:00
f063f02c41
solana-test-validator now uses the BPF JIT by default, `--no-bpf-jit` to disable
2021-02-09 20:24:14 +00:00
2caed6bd4c
Complete --rpc-pubsub-max-in-buffer-capacity/--rpc-pubsub-max-out-buffer-capacity plumbing
2021-02-09 07:24:30 +00:00
b6f231b60e
removes locked pubkey references ( #15152 )
2021-02-08 02:07:00 +00:00
bbae23358c
ledger-tool cleanup and additions ( #15179 )
...
* Plumb allow-dead-slots to ledger-tool verify
* ledger-tool cleanup and add some useful missing args
Print root slots and how many unrooted past last root.
2021-02-06 17:26:42 -08:00
fbf9dc47e9
Threadpool2 ( #15151 )
...
* rework thread pool for hash calculation
* rename
2021-02-05 18:48:16 -06:00
fabecdc86c
use thread pool for non-index hash calculations ( #15149 )
2021-02-05 19:48:55 +00:00
6fd5ec0e4c
caches descendants in bank forks ( #15107 )
2021-02-05 18:00:45 +00:00
c69027e5aa
chore: bump serde from 1.0.118 to 1.0.122 ( #15126 )
...
* chore: bump serde from 1.0.118 to 1.0.122
* Update programs/bpf/Cargo.lock
2021-02-05 19:23:24 +09:00
d1563f0ccd
Bump tonic, prost, tarpc, tokio ( #15013 )
...
* Update tonic & prost, and regenerate proto
* Reignore doc code
* Revert pull #14367 , but pin tokio to v0.2 for jsonrpc
* Bump backoff and goauth -> and therefore tokio
* Bump tokio in faucet, net-utils
* Bump remaining tokio, plus tarpc
2021-02-05 00:21:53 -07:00
600ff0d915
calculate hash from store instead of index ( #15034 )
...
* calculate hash from store instead of index
* restore update hash in abs
2021-02-04 09:00:33 -06:00
86467d825a
removes pubkey references ( #15050 )
2021-02-03 23:02:11 +00:00
0ad063f4e9
adds flag to disable duplicate instance check ( #15006 )
2021-02-03 16:26:17 +00:00
98aa1fa4ea
Upgrade jsonrpc crates to v17.0.0 ( #15018 )
...
* Upgrade to jsonrpc 17.0.0
* Fix test
* tree
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-02 19:53:08 -07:00
1df93fa2be
chore: bump serde from 1.0.112 to 1.0.118 ( #14828 )
...
* chore: bump serde from 1.0.112 to 1.0.122
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.112 to 1.0.122.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.122 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Update frozen_abi digest following serde update
* Revert "chore: bump serde from 1.0.112 to 1.0.122"
This reverts commit a3ef4442a4c985144ae2bd7ceaf8899a7ab8d7c0.
* Revert "[auto-commit] Update all Cargo lock files"
This reverts commit c41c3b005fb1ccade55155302c52cd5736c4b55f.
* chore: bump serde from 1.0.112 to 1.0.118
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.112 to 1.0.118.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.118 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Remove serum-dex pinning
* blind commit!
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-02-02 23:28:16 +09:00
cbb8b79a60
Add validator flag to opt in to cpi and logs storage ( #14922 )
...
* Add validator flag to opt in to cpi and logs storage
* Default TestValidator to opt-in; allow using in multinode-demo
* No clone
Co-authored-by: Carl Lin <carl@solana.com>
2021-02-01 14:00:51 -07:00
01230a0105
Remove serial_test_derive dependency ( #14891 )
2021-01-28 22:35:31 -07:00
d6873b82ab
Remove potentially too costly Packets::default() ( #14821 )
...
* Remove potentially too costly Packets::default()
* Fix test...
* Restore Packets::default()
* Restore Packets::default() more
2021-01-29 09:32:38 +09:00
119e2c75dd
cli now supports a custodian for stake authorize operations
2021-01-26 11:48:28 -08:00
ffa5c7dcc8
Deprecate commitment variants ( #14797 )
...
* Deprecate commitment variants
* Add new CommitmentConfig builders
* Add helpers to avoid allowing deprecated variants
* Remove deprecated transaction-status code
* Include new commitment variants in runtime commitment; allow deprecated as long as old variants persist
* Remove deprecated banks code
* Remove deprecated variants in core; allow deprecated in rpc/rpc-subscriptions for now
* Heavier hand with rpc/rpc-subscription commitment
* Remove deprecated variants from local-cluster
* Remove deprecated variants from various tools
* Remove deprecated variants from validator
* Update docs
* Remove deprecated client code
* Add new variants to cli; remove deprecated variants as possible
* Don't send new commitment variants to old clusters
* Retain deprecated method in test_validator_saves_tower
* Fix clippy matches! suggestion for BPF solana-sdk legacy compile test
* Refactor node version check to handle commitment variants and transaction encoding
* Hide deprecated variants from cli help
* Add cli App comments
2021-01-26 19:23:07 +00:00
d1df9da7d3
fixes test_filter_current flakiness ( #14816 )
2021-01-25 15:57:46 +00:00
bd0433c373
Clean up VerifiedVotePackets ( #14822 )
2021-01-25 09:01:47 +00:00
e1021d9f83
removes redundant epoch stakes cache in retransmit ( #14781 )
...
Following d6d76219b
2021-01-24 21:15:09 +00:00
491b059755
broadcasts duplicate shreds through gossip ( #14699 )
2021-01-24 15:47:43 +00:00
cbffab7850
Upgrade to Rust v1.49.0
2021-01-23 19:16:36 -08:00
bf1943e489
Add solana-test-validator --warp-slot argument
2021-01-22 21:17:02 -08:00
cbb9ac19b9
Add ability to clone accounts from an RPC endpoint
2021-01-22 13:29:36 -08:00
71e9958e06
Rpc: Add custom error for BigTable data not found ( #14762 )
...
* Expose not-found bigtable error
* Add custom rpc error for bigtable data not found
* Return custom rpc error when bigtable block is not found
* Generalize long-term storage
2021-01-22 04:40:47 +00:00
e4da6761a7
fixes test_filter_current flakiness ( #14749 )
2021-01-21 21:53:10 +00:00
8e581601d6
patches crds vote-index assignment bug ( #14438 )
...
If tower is full, old votes are evicted from the front of the deque:
https://github.com/solana-labs/solana/blob/2074e407c/programs/vote/src/vote_state/mod.rs#L367-L373
whereas recent votes if expire are evicted from the back:
https://github.com/solana-labs/solana/blob/2074e407c/programs/vote/src/vote_state/mod.rs#L529-L537
As a result, from a single tower_index scalar, we cannot infer which crds-vote
should be overwritten:
https://github.com/solana-labs/solana/blob/2074e407c/core/src/crds_value.rs#L576
In addition there is an off by one bug in the existing code. tower_index is
bounded by MAX_LOCKOUT_HISTORY - 1:
https://github.com/solana-labs/solana/blob/2074e407c/core/src/consensus.rs#L382
So, it is at most 30, whereas MAX_VOTES is 32:
https://github.com/solana-labs/solana/blob/2074e407c/core/src/crds_value.rs#L29
Which means that this branch is never taken:
https://github.com/solana-labs/solana/blob/2074e407c/core/src/crds_value.rs#L590-L593
so crds table alwasys keeps 29 **oldest** votes by wallclock, and then
only overrides the 30st one each time. (i.e a tally of only two most
recent votes).
2021-01-21 13:08:07 +00:00
04ce33a04e
Ensure sanitary transactions
2021-01-20 23:59:32 -08:00
dcaa025822
Configure Bigtable's timeout, enabling by default ( #14657 )
...
* Configure bigtable's timeout when read-only
* Review comments
* Apply nits (thanks!)
Co-authored-by: Michael Vines <mvines@gmail.com>
* Timeout in the streamed decoding as well
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-01-19 13:57:16 +00:00
b5fd0ed859
rewrites turbine retransmit peers computation ( #14584 )
2021-01-19 04:18:47 +00:00
c6ae0667e6
feature gates turbine retransmit peers patch ( #14631 )
2021-01-19 04:16:19 +00:00
5d9dc609b1
Rename RpcNodeUnhealthy error to NodeUnhealthy, generalize `getHealth` RPC error object for the future
2021-01-18 19:57:25 -08:00
cbf8ef7480
Make Bigtable::get_confirmed_blocks inclusive of requested start_slot and end_slot ( #14651 )
...
* Fix off-by-one error
* Filter out blocks greater than end slot
2021-01-18 19:14:10 -07:00
5f14f45282
More generic accounts purge functions ( #14595 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-17 20:31:03 -08:00
8d4ab1bab1
Clean up and add comment
2021-01-17 19:59:59 -08:00
5cf9094bb9
WIP fix the occasional stuck RPC request
2021-01-17 19:59:59 -08:00
4003f86f04
Add `getSnapshotSlot` RPC method
2021-01-16 19:31:21 +00:00
dacb95083d
Add `getHealth` RPC method
2021-01-16 10:10:48 -08:00
9a89689ad3
Use optimistic confirmation in getSignatureStatuses, and various downstream client methods ( #14430 )
...
* Add optimistically_confirmed field to TransactionStatus
* Update docs
* Convert new field to confirmation_status
* Update docs to confirmationStatus
* Update variants
* Update docs
* Just Confirmed
2021-01-15 16:05:05 +00:00
4ecc0f4fa6
chore: bump rayon from 1.4.1 to 1.5.0 ( #14177 )
...
* chore: bump rayon from 1.4.1 to 1.5.0
Bumps [rayon](https://github.com/rayon-rs/rayon ) from 1.4.1 to 1.5.0.
- [Release notes](https://github.com/rayon-rs/rayon/releases )
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md )
- [Commits](https://github.com/rayon-rs/rayon/compare/rayon-core-v1.4.1...rayon-core-v1.5.0 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-01-15 18:07:35 +09:00
1c2ae15b1d
Improve solana-test-validator output
2021-01-14 23:07:24 -08:00
907f518f6d
Add load/execute/store timings ( #14561 )
2021-01-14 14:14:16 -08:00
cfcca1cd3c
patches bug in turbine's neighbors computation ( #14565 )
...
Removing local node's index early from the set here:
https://github.com/solana-labs/solana/blob/e1b59ded4/core/src/retransmit_stage.rs#L346
distorts the order of nodes depending on which node is computing the
turbine fan-out tree, and results in incorrect neighbors computation.
2021-01-13 22:25:29 +00:00
11daaadc93
Add --rpc-threads argument
2021-01-13 13:34:46 -08:00
e1b59ded4b
Use leader_forward_count for tx retries too ( #14547 )
2021-01-13 11:14:22 -07:00
935dfdf0f6
fill in timing gaps in replay_stage ( #14550 )
...
* fill in timing gaps in replay_stage
* add replay_stage bank_count metric
* formatting
* handle another gap
* cleanup wait_receive_time to be more straightforward
2021-01-13 10:08:53 -06:00
6dfad0652f
Cache account stores, flush from AccountsBackgroundService ( #13140 )
2021-01-11 17:00:23 -08:00
b0e6e29527
Update timestamp max allowable drift to 50% of PoH ( #14531 )
...
* Repurpose warp-timestamp feature for general bump
* Change max_allowable_drift to 50%
* Fill in PR#
* Fix rpc test setup
2021-01-11 23:27:30 +00:00
a95675a7ce
Avoid tmp snapshot backlog in SnapshotPackagerService under high load ( #14516 )
2021-01-11 10:21:15 -08:00
7be6770808
Rename CompressionType to ArchiveFormat
2021-01-09 09:07:49 -08:00
141e6706e6
Rename AccountsPackage::root to AccountsPackage::slot
2021-01-09 09:07:49 -08:00
766195dded
limits number of crds values associated with a pubkey ( #14467 )
2021-01-08 18:54:40 +00:00
773b21b34e
consolidate DEFAULT_HASHES_PER_TICK ( #14463 )
2021-01-07 09:49:42 -06:00
c282586753
Add fixed tick rate adjustment ( #14447 )
2021-01-05 19:25:44 -08:00
ce1766d798
Enable program-id account index for supply calculations ( #14444 )
...
* Enable program-id account index for supply calculations
* Fixup comments
2021-01-06 02:42:33 +00:00
5affd8aa72
Add secondary indexes ( #14212 )
2020-12-31 18:06:03 -08:00
4a3d217839
Only initialize BigTable upload service when requested
2020-12-31 17:40:51 -08:00
0b23abd479
Bind RPC and faucet to 0.0.0.0
2020-12-31 07:55:26 +00:00
3d077fb656
Revert "Upgrade in-tree tokio 0.2 usage to tokio 0.3"
...
This reverts commit 444ed768dc
2020-12-31 06:26:53 +00:00
fb6c660cfd
Port ip-echo-server to tokio 0.3
2020-12-30 09:29:16 -08:00
fe667db910
validator: Add experimental flag to select PoH pinned core
2020-12-29 19:15:44 -07:00
82f61c0c4a
core: Update stale error message
2020-12-29 19:15:44 -07:00
f8a4afc7c1
Fix flaky broadcast test ( #14329 )
2020-12-29 12:35:04 -08:00
2074e407cd
Add poh speed check and tick speed calibration ( #14292 )
2020-12-29 09:35:57 -08:00
444ed768dc
Upgrade in-tree tokio 0.2 usage to tokio 0.3
2020-12-29 09:23:23 -08:00
3f10fb993b
Retry durable-nonce transactions ( #14308 )
...
* Retry durable-nonce transactions
* Add metric to track durable-nonce txs in queue
* Populate send-tx-service initial addresses with tpu_address if empty (primarily for testing)
* Reinstate last_valid_slot check for durable-nonce txs; use arbitrary future slot
2020-12-29 09:48:43 -07:00
9ddd6f08e8
Persist gossip contact info
2020-12-27 20:46:54 -08:00
c693ffaa08
Fix subtraction overflow in metrics ( #14290 )
2020-12-27 16:26:22 -08:00
2fd38d9912
indexes votes in crds table ( #14272 )
2020-12-27 13:31:05 +00:00
49019c6613
obtains staked-nodes from the root-bank ( #14257 )
...
... as opposed to the working bank
2020-12-27 13:28:05 +00:00
5b903318b2
vote: Add helper for creating current-versioned states
2020-12-22 19:37:26 -07:00
ace360ade2
Multiple entrypoint support
2020-12-22 18:35:31 -08:00
3373082ffa
Update entrypoint contact info even when shred version adoption is not requested
2020-12-22 18:35:31 -08:00
a14cfd660a
removes &Arc<Self> receivers ( #14234 )
2020-12-22 23:51:53 +00:00
baa9602411
Add shrink paths ( #14238 )
2020-12-21 21:33:37 -08:00
7b08cb1f0d
improves performance in replay-stage ( #14217 )
...
bank::vote_accounts returns a hash-map which is slow to iterate, but all uses
only require an iterator:
https://github.com/solana-labs/solana/blob/b3dc98856/runtime/src/bank.rs#L4300-L4306
Similarly, calculate_stake_weighted_timestamp takes a hash-map whereas it only
requires an iterator:
https://github.com/solana-labs/solana/blob/b3dc98856/sdk/src/stake_weighted_timestamp.rs#L21-L28
2020-12-21 19:18:19 +00:00
Jeff Washington (jwash)
Lijun Wang
dependabot[bot]
Michael Vines
Trent Nelson
Jon Cinque
behzad nouri
carllin
sakridge
Tao Zhu
Justin Starry
Tyera Eulberg
Ryo Onodera
Nikita
publish-docs.sh
steviez
Christian Drappi
François Garillot
Josh
Tyera Eulberg
Greg Fitzgerald
Ivan Mironov
Tom Parker-Shemilt
carllin