blockworks-foundation
/
solana
mirror of https://github.com/blockworks-foundation/solana.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
17,472 Commits 168 Branches 497 Tags 539 MiB
Commit Graph

2773 Commits

Author SHA1 Message Date
Michael 4ff50519ff
Add an info log to indicate the node has reached supermajority and print the active stake percentage (#19893) 2021-09-14 21:48:15 -06:00
Jeff Washington (jwash) b57e86abf2
cache account hash info (#19426) 
* cache account hash info

* ledger_path -> accounts_hash_cache_path
 2021-09-13 20:39:26 -05:00
carllin 87a7f00926
Track reset bank in PohRecorder (#19810) 2021-09-13 16:55:35 -07:00
Brooks Prumo 62c8bcf565
Add default() to SnapshotConfig (#19776) 2021-09-12 13:44:27 -05:00
Giles Cope c9261e034e remove dev deps 2021-09-11 14:33:53 -07:00
Giles Cope c3a610374a unused deps 2021-09-11 14:33:53 -07:00
Brooks Prumo 7aa5f6b833
Add CLI args for incremental snapshots (#19694) 
Add `--incremental-snapshots` flag to enable incremental snapshots.
This will allow setting `--full-snapshot-interval-slots` and
`--incremental-snapshot-interval-slots`.

Also added `--maximum-incremental-snapshots-to-retain`.

Co-authored-by: Michael Vines <mvines@gmail.com>
 2021-09-10 15:59:26 -05:00
Michael Vines 4386e09710 Reduce wait for supermajority threshold back to 80% 2021-09-09 21:17:35 -07:00
sakridge 3a8c678f62
Remove some copying (#19691) 2021-09-08 18:32:38 +02:00
Jeff Washington (jwash) 456bf15012
AccountsIndexConfig -> AccountsDbConfig (#19687) 2021-09-08 04:30:38 +00:00
Jeff Washington (jwash) d3f938f0cf
Remove Copy from AccountsIndexConfig. Not all types will support it (#19686) 2021-09-07 20:09:40 -05:00
Brooks Prumo 4a5f83d3a7
Add fn to check when to take snapshots (#19682) 2021-09-07 23:26:35 +00:00
Brooks Prumo a0552e5b46
Make startup aware of Incremental Snapshots (#19600) 2021-09-07 20:43:43 +00:00
behzad nouri 01a7ec8198
uses rayon thread-pool for retransmit-stage parallelization (#19486) 2021-09-07 15:15:01 +00:00
Brooks Prumo fe8ba81ce6
Rename to is_valid instead of is_invalid (#19670) 2021-09-07 09:31:54 -05:00
dependabot[bot] fdbde2df52
chore: bump etcd-client from 0.7.1 to 0.7.2 (#19661) 
Bumps [etcd-client](https://github.com/etcdv3/etcd-client) from 0.7.1 to 0.7.2.
- [Release notes](https://github.com/etcdv3/etcd-client/releases)
- [Commits](https://github.com/etcdv3/etcd-client/compare/0.7.1...0.7.2)

---
updated-dependencies:
- dependency-name: etcd-client
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-06 19:53:28 -06:00
Brooks Prumo 9d9482b9d8
Plumb `maximum_incremental_snapshot_archives_to_retain` (#19640) 2021-09-06 18:01:56 -05:00
Sean Young d461a9ac10 verify_precompiles needs FeatureSet 
Rather than pass in individual features, pass in the entire feature set
so that we can add the ed25519 program feature in a later commit.
 2021-09-05 18:59:37 +01:00
Tyera Eulberg decec3cd8b
Demote write locks on transaction program ids (#19593) 
* Add feature

* Demote write lock on program ids

* Fixup bpf tests

* Update MappedMessage::is_writable

* Comma nit

* Review comments
 2021-09-04 03:05:30 +00:00
Brooks Prumo 1828579580
Pass SnapshotConfig to SnapshotPackagerService (#19616) 2021-09-03 21:42:32 +00:00
Brooks Prumo 5e25ee5ebe
Add maximum_incremental_snapshot_archives_to_retain to SnapshotConfig (#19612) 2021-09-03 20:21:32 +00:00
Brooks Prumo 7ab0aec61f
Rename maximum_full_snapshot_archives_to_retain (#19610) 
To prepare for adding maximum_incremental_snapshot_archives_to_retain,
rename the current field in SnapshotConfig.
 2021-09-03 11:28:10 -05:00
Brooks Prumo e9374d32a3
Revert "Make startup aware of Incremental Snapshots (#19550)" (#19599) 
This reverts commit d45ced0a5d.
 2021-09-02 19:14:41 -05:00
Brooks Prumo d45ced0a5d
Make startup aware of Incremental Snapshots (#19550) 2021-09-02 19:05:15 -05:00
Jeff Biseda 7a8eba10b2
add synchronization comment to handle_new_root (#19571) 2021-09-02 13:52:14 -07:00
Lijun Wang 8378e8790f
Accountsdb replication installment 2 (#19325) 
This is the 2nd installment for the AccountsDb replication.

Summary of Changes

The basic google protocol buffer protocol for replicating updated slots and accounts. tonic/tokio is used for transporting the messages.

The basic framework of the client and server for replicating slots and accounts -- the persisting of accounts in the replica-side will be done at the next PR -- right now -- the accounts are streamed to the replica-node and dumped. Replication for information about Bank is also not done in this PR -- to be addressed in the next PR to limit the change size.

Functionality used by both the client and server side are encapsulated in the replica-lib crate.

There is no impact to the existing validator by default.

Tests:

Observe the confirmed slots replicated to the replica-node.
Observe the accounts for the confirmed slot are received at the replica-node side.
 2021-09-01 14:10:16 -07:00
behzad nouri 6d9818b8e4
skips retransmit for shreds with unknown slot leader (#19472) 
Shreds' signatures should be verified before they reach retransmit
stage, and if the leader is unknown they should fail signature check.
Therefore retransmit-stage can as well expect to know who the slot
leader is and otherwise just skip the shred.

Blockstore checking signature of recovered shreds before sending them to
retransmit stage:
https://github.com/solana-labs/solana/blob/4305d4b7b/ledger/src/blockstore.rs#L884-L930

Shred signature verifier:
https://github.com/solana-labs/solana/blob/4305d4b7b/core/src/sigverify_shreds.rs#L41-L57
https://github.com/solana-labs/solana/blob/4305d4b7b/ledger/src/sigverify_shreds.rs#L105
 2021-09-01 15:44:26 +00:00
Brooks Prumo 1d5a8ebc6a
Revert "Add LastFullSnapshotSlot to SnapshotConfig (#19341)" (#19529) 
This reverts commit 4d361af976.
 2021-08-31 22:03:19 -05:00
Brooks Prumo fe9ee9134a
Make background services aware of incremental snapshots (#19401) 
AccountsBackgroundService now knows about incremental snapshots.  It is
now also in charge of deciding if an AccountsPackage is destined to be a
SnapshotPackage or not (or just used by AccountsHashVerifier).

!!! New behavior changes !!!

Taking snapshots (both bank and archive) **MUST** succeed.

This is required because of how the last full snapshot slot is
calculated, which is used by AccountsBackgroundService when calling
`clean_accounts()`.

File system calls are now unwrapped and will result in a crash. As Trent told me:

>Well I think if a snapshot fails due to some IO error, it's very likely that the operator is going to have to intervene before it works.  We should exit error in this case, otherwise the validator might happily spin for several more hours, never successfully writing a complete snapshot, before something else brings it down.  This would leave the validator's last local snapshot many more slots behind than it would be had we exited outright and potentially force the operator to abandon ledger continuity in favor of a quick catchup

Other errors will set the `exit` flag to `true`, and the node will gracefully shutdown.

Fixes #19167 
Fixes #19168
 2021-08-31 18:33:27 -05:00
dependabot[bot] c21bf29ce7
chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.66 to 1.0.67.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.66...v1.0.67)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-31 06:05:25 +00:00
Tyera Eulberg a3bef2e537
Fix shreds-to-hours/days estimations (#19477) 2021-08-30 13:16:06 -06:00
dependabot[bot] 6a53ec28e0
chore: bump serde from 1.0.129 to 1.0.130 (#19497) 
* chore: bump serde from 1.0.129 to 1.0.130

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.129 to 1.0.130.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.129...v1.0.130)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-08-30 18:19:34 +00:00
behzad nouri 8ad52fa095
implements copy-on-write for vote-accounts (#19362) 
Bank::vote_accounts redundantly clones vote-accounts HashMap even though
an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/95c998a19/runtime/src/bank.rs#L5174-L5186

This commit implements copy-on-write semantics for vote-accounts by
wrapping the underlying HashMap in Arc<...>.
 2021-08-30 15:54:01 +00:00
carllin 84db04ce6c
Fix duplicate broadcast test (#19365) 2021-08-27 17:53:24 -07:00
Justin Starry 2d7f036afd
Add solana-program-runtime crate (#19438) 2021-08-27 00:30:36 +00:00
dependabot[bot] e492638b7d
chore: bump libc from 0.2.100 to 0.2.101 (#19442) 
* chore: bump libc from 0.2.100 to 0.2.101

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.100 to 0.2.101.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.100...0.2.101)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
 2021-08-26 12:38:34 -06:00
dependabot[bot] 46890ac197
chore: bump serde from 1.0.128 to 1.0.129 (#19395) 
* chore: bump serde from 1.0.128 to 1.0.129

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.128 to 1.0.129.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.128...v1.0.129)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-08-24 12:32:30 -06:00
dependabot[bot] c0c95e88d8
chore: bump libc from 0.2.99 to 0.2.100 (#19371) 
* chore: bump libc from 0.2.99 to 0.2.100

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.99 to 0.2.100.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.99...0.2.100)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-08-23 09:34:32 -06:00
dependabot[bot] aea9960c75
chore: bump serde from 1.0.127 to 1.0.128 (#19370) 
* chore: bump serde from 1.0.127 to 1.0.128

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.127 to 1.0.128.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.127...v1.0.128)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
 2021-08-23 09:33:34 -06:00
Brooks Prumo 1a4bede918
Remove filter_snapshot_storages_for_incremental_snapshot() (#19349) 
Now that we can get just the subset of snapshot storages we want, the extra
filtering afterwards is redundant.
 2021-08-23 09:45:29 -05:00
Brooks Prumo 6d939811e9
Name snapshots consistently (#19346) 
#### Problem

Snapshot names are overloaded, and there are multiple terms that mean the same thing. This is confusing. Here's a list of ones in the codebase that I've found:

```
- snapshot_dir
- snapshots_dir
- snapshot_path
- snapshot_output_dir
- snapshot_package_output_path
- snapshot_archives_dir
```

#### Summary of Changes

For all the ones that are about the directory where snapshot archives are stored, ensure they are `snapshot_archives_dir`. For the ones about the (bank) snapshots directory, set to `bank_snapshots_dir`.


Co-authored-by: Michael Vines <mvines@gmail.com>
 2021-08-21 15:41:03 -05:00
Brooks Prumo 234461f779
Add base_slot to get_snapshot_storages() (#19348) 2021-08-20 16:23:43 -05:00
Brooks Prumo 4d361af976
Add LastFullSnapshotSlot to SnapshotConfig (#19341) 2021-08-20 17:06:53 +00:00
behzad nouri 1deb4add81
removes Slot from TransmitShreds (#19327) 
An earlier version of the code was funneling through stakes along with
shreds to broadcast:
https://github.com/solana-labs/solana/blob/b67ffab37/core/src/broadcast_stage.rs#L127

This was changed to only slots as stakes computation was pushed further
down the pipeline in:
https://github.com/solana-labs/solana/pull/18971

However shreds themselves embody which slot they belong to. So pairing
them with slot is redundant and adds rooms for bugs should they become
inconsistent.
 2021-08-20 13:48:33 +00:00
Trent Nelson e0bc5fa690 validator: Trusted validators are now called known validators 2021-08-19 22:43:49 -06:00
Jack May 3ec33e7d02
Fail secp256k1 if the instruction data looks incorrect (#19300) 2021-08-19 13:13:54 -07:00
Tao Zhu 4982dc20f9
replace function with const var for better readability (#19285) 2021-08-19 14:59:53 -05:00
Justin Starry c50b01cb60
Store versioned transactions in the ledger, disabled by default (#19139) 
* Add support for versioned transactions, but disable by default

* merge conflicts

* trent's feedback

* bump Cargo.lock

* Fix transaction error encoding

* Rename legacy_transaction method

* cargo clippy

* Clean up casts, int arithmetic, and unused methods

* Check for duplicates in sanitized message conversion

* fix clippy

* fix new test

* Fix bpf conditional compilation for message module
 2021-08-17 15:17:56 -07:00
Jeff Washington (jwash) 7c70f2158b
accounts_index_bins to AccountsIndexConfig (#19257) 
* accounts_index_bins to AccountsIndexConfig

* rename param bins -> config

* rename BINS_FOR* to ACCOUNTS_INDEX_CONFIG_FOR*
 2021-08-17 14:50:01 -05:00
Brooks Prumo f9986c66b8
Make SnapshotPackagerService aware of Incremental Snapshots (#19254) 
Add a field to SnapshotPackage that is an enum for SnapshotType, so archive_snapshot_package() will do the right thing.

Fixes #19166
 2021-08-17 13:01:59 -05:00
behzad nouri 7a8807b8bb retransmits shreds recovered from erasure codes 
Shreds recovered from erasure codes have not been received from turbine
and have not been retransmitted to other nodes downstream. This results
in more repairs across the cluster which is slower.

This commit channels through recovered shreds to retransmit stage in
order to further broadcast the shreds to downstream nodes in the tree.
 2021-08-17 13:44:10 +00:00
behzad nouri 3efccbffab sends shreds (instead of packets) to retransmit stage 
Working towards channelling through shreds recovered from erasure codes
to retransmit stage.
 2021-08-17 13:44:10 +00:00
behzad nouri 6e413331b5 removes erroneous uses of Arc<...> from retransmit stage 2021-08-17 13:44:10 +00:00
behzad nouri 8198a7eae1 adds packet/shred count stats to window-service 
Adding back these metrics from the earlier commit which removed them
from retransmit stage.
 2021-08-17 13:44:10 +00:00
behzad nouri bf437b0336 removes packet-count metrics from retransmit stage 
Working towards sending shreds (instead of packets) to retransmit stage
so that shreds recovered from erasure codes are as well retransmitted.

Following commit will add these metrics back to window-service, earlier
in the pipeline.
 2021-08-17 13:44:10 +00:00
behzad nouri 563aec0b4d
discards epoch-slots epochs ahead of the current root (#19256) 
Cross cluster gossip contamination is causing cluster-slots hash map to
contain a lot of bogus values and consume too much memory:
https://github.com/solana-labs/solana/issues/17789

If a node is using the same identity key across clusters, then these
erroneous values might not be filtered out by shred-versions check,
because one of the variants of the contact-info will have matching
shred-version:
https://github.com/solana-labs/solana/issues/17789#issuecomment-896304969

The cluster-slots hash-map is bounded and trimmed at the lower end by
the current root. This commit also discards slots epochs ahead of the
root.
 2021-08-17 13:13:28 +00:00
behzad nouri f33b7abffb
adds back cluster partitions to broadcast-duplicates (#19253) 
An earlier version of this code was aiming to create a partition by
manipulating stakes, and setting some of them to zero:
https://github.com/solana-labs/solana/blob/cde146155/core/src/broadcast_stage/broadcast_duplicates_run.rs#L65-L116

https://github.com/solana-labs/solana/pull/18971
moved stakes computation further down the stream, and so that logic
could no longer live there. This commit adds back cluster partitions
by intercepting packets before send.
 2021-08-16 22:24:30 +00:00
Michael Vines 3e5ba594e0 Revert `TestValidatorGenesis::start()` to v1.7.8 signature; add `TestValidatorGenesis::start_with_socket_addr_space()` 2021-08-16 06:37:23 +00:00
Michael Vines b15fa9fbd2 Add EtcdTowerStorage 2021-08-14 09:46:36 -07:00
carllin 22674000bd
Add EpochSlots frozen state transition (#19112) 2021-08-13 14:21:52 -07:00
Brooks Prumo 176036aa58
Rename AccountsPacakge to SnapshotPackage and AccountsPackagePre to AccountsPackage (#19231) 
Renaming these types to better communicate their usages, which will
further diverge as incremental snapshot support is added.

With the new names, AccountsPacakge now refers to the type between
AccountsBackgroundProcess and AccountsHashVerifier, and SnapshotPackage
refers to the type between AccountsHashVerifier and
SnapshotPackagerService.
 2021-08-13 16:08:09 -05:00
behzad nouri b64eeb7729 removes erroneous uses of &Arc<...> from window-service 2021-08-13 17:26:31 +00:00
behzad nouri d57398a959 removes repeated bank-forks locking in window-service 
Window service is repeatedly locking bank-forks to look-up working-bank
for every single shred:
https://github.com/solana-labs/solana/blob/5fde4ee3a/core/src/window_service.rs#L597-L606

This commit updates shred_filter signature in recv_window so that where
we already obtain the lock on bank-forks, we can also look-up
working-bank once for all packets:
https://github.com/solana-labs/solana/blob/5fde4ee3a/core/src/window_service.rs#L256-L277
 2021-08-13 17:26:31 +00:00
dependabot[bot] 5b9671d01a
chore: bump matches from 0.1.8 to 0.1.9 (#19220) 
* chore: bump matches from 0.1.8 to 0.1.9

Bumps [matches](https://github.com/SimonSapin/rust-std-candidates) from 0.1.8 to 0.1.9.
- [Release notes](https://github.com/SimonSapin/rust-std-candidates/releases)
- [Commits](https://github.com/SimonSapin/rust-std-candidates/commits)

---
updated-dependencies:
- dependency-name: matches
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-08-13 16:37:45 +00:00
Jack May 0b50bb2b20
Deprecate FeeCalculator returning APIs (#19120) 2021-08-13 09:08:20 -07:00
behzad nouri 7a789e0763
filters for recent contact-infos when checking for live stake (#19204) 
Contact-infos are saved to disk:
https://github.com/solana-labs/solana/blob/9dfeee299/gossip/src/cluster_info.rs#L1678-L1683

and restored on validator start-up:
https://github.com/solana-labs/solana/blob/9dfeee299/core/src/validator.rs#L450

Staked nodes entries will not expire until an epoch after. So when the
validator checks for online stake it is erroneously picking up
contact-infos restored from disk, which breaks the entire
wait-for-supermajority logic:
https://github.com/solana-labs/solana/blob/9dfeee299/core/src/validator.rs#L1515-L1561

This commit adds an extra check for the age of contact-info entries and
filters out old ones.
 2021-08-13 12:12:40 +00:00
Tao Zhu 414d904959
Reject blocks for costs above the max block cost (#18994) 
* added realtime cost checking logic to reject block that would exceed max limit:
- defines max limits at block_cost_limits.rs
- right after each bath's execution, accumulate its cost and check again
  limit, return error if limit is exceeded

* update abi that changed due to adding additional TransactionError

* To avoid counting stats mltiple times, only accumulate execute-timing when a bank is completed

* gate it by a feature

* move cost const def into block_cost_limits.rs

* redefine the cost for signature and account access, removed signer part as it is not well defined for now

* check if per_program_timings of execute_timings before sending
 2021-08-12 10:48:47 -05:00
Brooks Prumo 9d8594a046
Add test for snapshots with background services (#19158) 
Add a test for snapshots that spins up AccountsBackgroundService,
AccountsHashVerifier, and SnapshotPackagerService.

Currently there is not a test for snapshots that spins up the background
services fully.  This means that there's not a current test that I can
use when adding incremental snapshot support to these three services.

Fixes #19014
 2021-08-12 10:18:20 -05:00
Jeff Washington (jwash) e91988c977
cli for num account index bins (#19085) 2021-08-11 11:45:25 -05:00
Michael Vines 7ddda30126 `solana-test-validator` now uses FileTowerStorage 2021-08-11 00:20:46 -07:00
Michael Vines e9722474eb Move tower storage into its own module 2021-08-11 00:20:46 -07:00
Michael Vines d7ab510229 Move tower save into the VotingService 2021-08-11 00:20:46 -07:00
behzad nouri 00e5e12906 renames solana_runtime::vote_account::VoteAccount 
Rename:
  VoteAccount    -> VoteAccountInner  # the private type
  ArcVoteAccount -> VoteAccount       # the public type
 2021-08-10 22:54:17 +00:00
Brooks Prumo faf99f4760
Move Bank::get_incremental_snapshot_storages() into snapshot_utils (#19155) 
Filtering out storages for incremental snapshots will be needed by the
background services for incremental snapshot support, but there is not a
Bank at that point.  Since the filtering doesn't apply only to Bank, and
more to snapshots, move the functionality into snapshot_utils.
 2021-08-10 14:04:11 -05:00
Brooks Prumo ccfa82461b
Pass SnapshotConfig to AccountsHashVerifier (#19154) 
AccountsHashVerifier will need access to both the full and incremental
snapshot archive interval slots config values, which is in the
SnapshotConfig.

Also, cleanup some `Option<>` params and their references.
 2021-08-10 14:02:34 -05:00
dependabot[bot] 536b763751
chore: bump libc from 0.2.98 to 0.2.99 (#19142) 
* chore: bump libc from 0.2.98 to 0.2.99

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.98 to 0.2.99.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.98...0.2.99)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-08-10 10:54:54 -06:00
Jeff Washington (jwash) 651343688d
Use option for account index bucket parameter (#19150) 2021-08-10 11:32:25 -05:00
Jeff Washington (jwash) 47e0d9aa95
plumb more accounts_index bins (#19123) 2021-08-10 05:45:46 -05:00
Brian Anderson e4b66a5913
Fix deserialization of RPC errors in HttpSender (#19110) 
Fixes #15576
 2021-08-09 10:45:00 -07:00
Brooks Prumo fd937548a0
Move SnapshotArchiveInfo and friends into its own module (#19114) 2021-08-08 07:57:06 -05:00
Brooks Prumo 00890957ee
Add snapshot_utils::bank_from_latest_snapshot_archives() (#18983) 
While reviewing PR #18565, as issue was brought up to refactor some code
around verifying the bank after rebuilding from snapshots.  A new
top-level function has been added to get the latest snapshot archives
and load the bank then verify.  Additionally, new tests have been
written and existing tests have been updated to use this new function.

Fixes #18973

While resolving the issue, it became clear there was some additional
low-hanging fruit this change enabled.  Specifically, the functions
`bank_to_xxx_snapshot_archive()` now return their respective
`SnapshotArchiveInfo`.  And on the flip side,
`bank_from_snapshot_archives()` now takes `SnapshotArchiveInfo`s instead
of separate paths and archive formats.  This bundling simplifies bank
rebuilding.
 2021-08-06 20:16:06 -05:00
Michael Vines 397801a2d8 Extract tower storage details from Tower struct 2021-08-06 10:04:37 -07:00
Jeff Washington (jwash) ca37873e16
rework bank::new_with_paths (#19087) 
* rework bank::new_with_paths

* missing 1 bench
 2021-08-06 09:30:40 -05:00
behzad nouri e4be00fece falls back on working-bank if root-bank::epoch-staked-nodes is none 
bank.get_leader_schedule_epoch(shred_slot)
is one epoch after epoch_schedule.get_epoch(shred_slot).

At epoch boundaries, shred is already one epoch after the root-slot. So
we need epoch-stakes 2 epochs ahead of the root. But the root bank only
has epoch-stakes for one epoch ahead, and as a result looking up epoch
staked-nodes from the root-bank fails.

To be backward compatible with the current master code, this commit
implements a fallback on working-bank if epoch staked-nodes obtained
from the root-bank is none.
 2021-08-05 21:47:33 +00:00
behzad nouri eaf927cf49 allows only one thread to update cluster-nodes cache entry for an epoch 
If two threads simultaneously call into ClusterNodesCache::get for the
same epoch, and the cache entry is outdated, then both threads recompute
cluster-nodes for the epoch and redundantly overwrite each other.

This commit wraps ClusterNodesCache entries in Arc<Mutex<...>>, so that
when needed only one thread does the computations to update the entry.
 2021-08-05 21:47:33 +00:00
behzad nouri fb69f45f14 adds fallback & metric for when epoch staked-nodes are none 2021-08-05 21:47:33 +00:00
behzad nouri 50d0e830c9 unifies cluster-nodes computation & caching across turbine stages 
Broadcast-stage is using epoch_staked_nodes based on the same slot that
shreds belong to:
https://github.com/solana-labs/solana/blob/049fb0417/core/src/broadcast_stage/standard_broadcast_run.rs#L208-L228
https://github.com/solana-labs/solana/blob/0cf52e206/core/src/broadcast_stage.rs#L342-L349

But retransmit-stage is using bank-epoch of the working-bank:
https://github.com/solana-labs/solana/blob/19bd30262/core/src/retransmit_stage.rs#L272-L289

So the two are not consistent at epoch boundaries where some nodes may
have a working bank (or similarly a root bank) lagging other nodes. As a
result the node which obtains a packet may construct turbine broadcast
tree inconsistently with its parent node in the tree and so some packets
may fail to reach all nodes in the tree.
 2021-08-05 21:47:33 +00:00
behzad nouri aa32738dd5 uses cluster-nodes cache in broadcast-stage 
* Current caching mechanism does not update cluster-nodes when the epoch
  (and so epoch staked nodes) changes:
  https://github.com/solana-labs/solana/blob/19bd30262/core/src/broadcast_stage/standard_broadcast_run.rs#L332-L344

* Additionally, the cache update has a concurrency bug in which the
  thread which does compare_and_swap may be blocked when it tries to
  obtain the write-lock on cache, while other threads will keep running
  ahead with the outdated cache (since the atomic timestamp is already
  updated).

In the new ClusterNodesCache, entries are keyed by epoch, and so if
epoch changes cluster-nodes will be recalculated. The time-to-live
eviction policy is also encapsulated and rigidly enforced.
 2021-08-05 21:47:33 +00:00
behzad nouri 30bec3921e uses cluster-nodes cache in retransmit stage 
The new cluster-nodes cache will:
  * ensure cluster-nodes are recalculated if the epoch (and so the epoch
    staked nodes) changes.
  * encapsulate time-to-live eviction policy.
 2021-08-05 21:47:33 +00:00
behzad nouri ecc1c7957f implements cluster-nodes cache 
Cluster nodes are cached keyed by the respective epoch from which stakes
are obtained, and so if epoch changes cluster-nodes will be recomputed.

A time-to-live eviction policy is enforced to refresh entries in case
gossip contact-infos are updated.
 2021-08-05 21:47:33 +00:00
behzad nouri 44b11154ca sends slots (instead of stakes) through broadcast flow 
Current broadcast code is computing stakes for each slot before sending
them down the channel:
https://github.com/solana-labs/solana/blob/049fb0417/core/src/broadcast_stage/standard_broadcast_run.rs#L208-L228
https://github.com/solana-labs/solana/blob/0cf52e206/core/src/broadcast_stage.rs#L342-L349

Since the stakes are a function of epoch the slot belongs to (and so
does not necessarily change from one slot to another), forwarding the
slot itself would allow better caching downstream.

In addition we need to invalidate the cache if the epoch changes (which
the current code does not do), and that requires to know which slot (and
so epoch) current broadcasted shreds belong to:
https://github.com/solana-labs/solana/blob/19bd30262/core/src/broadcast_stage/standard_broadcast_run.rs#L332-L344
 2021-08-05 21:47:33 +00:00
Jeff Washington (jwash) e368f10973
add _for_tests to new_no_wallclock_throttle (#19086) 2021-08-05 14:50:25 -05:00
Jeff Washington (jwash) a9014ceceb
Bank::default_for_tests() (#19084) 2021-08-05 11:53:29 -05:00
behzad nouri 40914de811 updates cluster-slots with root-bank instead of root-slot + bank-forks 
ClusterSlots::update is taking both root-slot and bank-forks only to
later lookup root-bank from bank-forks, which is redundant. Also
potentially by the time bank-forks is locked to obtain root-bank,
root-slot may have already changed and so be inconsistent with the
root-slot passed in as the argument.
https://github.com/solana-labs/solana/blob/6d95d679c/core/src/cluster_slots.rs#L32-L39
https://github.com/solana-labs/solana/blob/6d95d679c/core/src/cluster_slots.rs#L122
 2021-08-05 14:43:06 +00:00
behzad nouri 2fc112edcf removes unused code from cluster-slots 2021-08-05 14:43:06 +00:00
Jeff Washington (jwash) bf16b0517c
add _for_tests to setup_bank_and_vote_pubkeys (#19060) 2021-08-05 08:43:35 -05:00
Jeff Washington (jwash) 14361906ca
for all tests, bank::new -> bank::new_for_tests (#19064) 2021-08-05 08:42:38 -05:00
Jeff Washington (jwash) bde9b4de94
Bank::new -> Bank::new_for_benches (#19063) 2021-08-04 17:30:43 -05:00
Jeff Washington (jwash) 3280ae3e9f
add validator option --accounts-db-skip-shrink (#19028) 
* add validator option --accounts-db-skip-shrink

* typo
 2021-08-04 17:28:33 -05:00
Jeff Washington (jwash) 1ed12a07ab
introduce Bank::new_for_tests (#19062) 2021-08-04 15:06:57 -05:00
Brooks Prumo ca14475085
Add incremental_snapshot_archive_interval_slots to SnapshotConfig (#19026) 
This commit also renames `snapshot_interval_slots` to
`full_snapshot_archive_interval_slots`, updates the comments on the
fields, and make appropriate updates where SnapshotConfig is used.
 2021-08-04 14:40:20 -05:00
Oliver 06e08c4840
move package_snapshots to AccountsPackagePre ctors (#18997) 
This PR solves #18815. Note that I had to make the snapshot prefix
constants inside `snapshot_utils.rs` public at the crate level in order
to make this work. I'm not sure whether or not introducing this
dependency is entirely good, either way the `snapshot_utils.rs` file
needs a lot of rework so things will move around, I believe this does
the work in the meantime. Any feedback will be greatly appreciated.
 2021-08-04 09:03:03 -05:00
Trent Nelson 06a7a9e544 remove superfluous `collect()`s 2021-08-04 07:21:55 +00:00
carllin 03353d500f
Actively manage dead slots in AncestorHashesService (#18912) 2021-08-02 14:33:28 -07:00
dependabot[bot] c16bf02448
chore: bump serde from 1.0.126 to 1.0.127 (#19010) 
* chore: bump serde from 1.0.126 to 1.0.127

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.126 to 1.0.127.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.126...v1.0.127)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-08-02 21:16:34 +00:00
behzad nouri 049fb0417f
allows sendmmsg api taking owned values (as well as references) (#18999) 
Current signature of api in sendmmsg requires a slice of inner
references:
https://github.com/solana-labs/solana/blob/fe1ee4980/streamer/src/sendmmsg.rs#L130-L152

That forces the call-site to convert owned values to references even
though doing so is redundant and adds an extra level of indirection:
https://github.com/solana-labs/solana/blob/fe1ee4980/core/src/repair_service.rs#L291

This commit expands the api using AsRef and Borrow traits to allow
calling the method with owned values (as well as references like
before).
 2021-07-30 20:58:49 +00:00
dependabot[bot] fe1ee49807
chore: bump serde_json from 1.0.65 to 1.0.66 (#18990) 
* chore: bump serde_json from 1.0.65 to 1.0.66

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.65 to 1.0.66.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.65...v1.0.66)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-07-30 10:46:47 -06:00
Tao Zhu 5d297ccf96
Cost model uses compute_unit to replace microsecond as cost unit (#18934) 
* wip - cost_update_services to log both us and cu for each instruction to determine possible ratio

* replace microsecond with compute_unit as cost unit
 2021-07-29 22:19:36 +00:00
Brooks Prumo b05fb87f22
Add test_bank_forks_incremental_snapshot() (#18565) 
This commit builds on PR #18504 by adding a test to core/tests/snapshot.rs for Incremental Snapshots. The test adds banks to bank forks in a loop and takes both full snapshots and incremental snapshots at intervals, and validates they are rebuild-able.

For background info about Incremental Snapshots, see #17088.

Fixes #18829 and #18972
 2021-07-29 16:46:54 -05:00
Ryo Onodera da480bdb5f
Fix unstable retransmit-num_nodes (#18970) 2021-07-29 17:32:32 +00:00
dependabot[bot] 0cf52e2060
chore: bump serde_json from 1.0.64 to 1.0.65 (#18968) 
* chore: bump serde_json from 1.0.64 to 1.0.65

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.64 to 1.0.65.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.64...v1.0.65)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-07-29 16:52:02 +00:00
behzad nouri d06dc6c8a6
shares cluster-nodes between retransmit threads (#18947) 
cluster_nodes and last_peer_update are not shared between retransmit
threads, as each thread have its own value:
https://github.com/solana-labs/solana/blob/65ccfed86/core/src/retransmit_stage.rs#L476-L477

Additionally, with shared references, this code:
https://github.com/solana-labs/solana/blob/0167daa11/core/src/retransmit_stage.rs#L315-L328
has a concurrency bug where the thread which does compare_and_swap,
updates cluster_nodes much later after other threads have run with
outdated cluster_nodes for a while. In particular, the write-lock there
may block.
 2021-07-29 16:20:15 +00:00
dependabot[bot] 1f788781dd
chore: bump raptorq from 1.6.3 to 1.6.4 (#18965) 
Bumps [raptorq](https://github.com/cberner/raptorq) from 1.6.3 to 1.6.4.
- [Release notes](https://github.com/cberner/raptorq/releases)
- [Commits](https://github.com/cberner/raptorq/compare/v1.6.3...v1.6.4)

---
updated-dependencies:
- dependency-name: raptorq
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-29 08:39:05 -06:00
dependabot[bot] 5f23d8530d
chore: bump lru from 0.6.5 to 0.6.6 (#18963) 
Bumps [lru](https://github.com/jeromefroe/lru-rs) from 0.6.5 to 0.6.6.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases)
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jeromefroe/lru-rs/compare/0.6.5...0.6.6)

---
updated-dependencies:
- dependency-name: lru
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-29 08:33:41 -06:00
Trent Nelson 71f6d839f9 validator: remove disused cuda config argument 2021-07-29 03:08:52 +00:00
Trent Nelson 8ed0cd0fff validator: check target CPU features earlier 2021-07-29 03:08:52 +00:00
Trent Nelson c435f7b3e3 validator: add avx2 runtime check 2021-07-29 03:08:52 +00:00
Trent Nelson e641f257ef test-validator: move feature check earlier in startup 2021-07-29 03:08:52 +00:00
Trent Nelson 59641623d1 Improve check for Apple M1 silicon under Rosetta 2021-07-29 03:08:52 +00:00
Jeff Biseda 9255ae334d
drop outstanding_requests lock before sending repair requests (#18893) 2021-07-28 19:30:43 -07:00
sakridge 84e78316b1
Write helper for multithread update (#18808) 2021-07-29 03:16:36 +02:00
dependabot[bot] 14f0ce850d
chore: bump blake3 from 0.3.8 to 1.0.0 (#18908) 
* chore: bump blake3 from 0.3.8 to 1.0.0

Bumps [blake3](https://github.com/BLAKE3-team/BLAKE3) from 0.3.8 to 1.0.0.
- [Release notes](https://github.com/BLAKE3-team/BLAKE3/releases)
- [Commits](https://github.com/BLAKE3-team/BLAKE3/compare/0.3.8...1.0.0)

---
updated-dependencies:
- dependency-name: blake3
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Make versions consistent

* Enable new feature

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-07-27 17:55:21 -06:00
Jack May f1b9f97aef
remove avx error on macos (#18923) 2021-07-27 16:34:04 -07:00
carllin c0704d4ec9
Plumb signal from replay to ancestor hashes service (#18880) 2021-07-26 20:59:00 -07:00
Tyera Eulberg 8596db8f53
Bump jsonrpc crates and remove old tokio (#18779) 
* Bump jsonrpc crates and replace old tokio

* Bump tokio

* getBlockTime

* getBlocks

* getBlocksWithLimit, getInflationReward

* getBlock

* getFirstAvailableBlock

* getTransaction

* getSignaturesForAddress

* getSignatureStatuses

* Remove superfluous runtime
 2021-07-26 12:32:17 -06:00
carllin 1ee64afb12
Introduce AncestorHashesService (#18812) 2021-07-23 16:54:47 -07:00
behzad nouri d2d5f36a3c
adds validator flag to allow private ip addresses (#18850) 2021-07-23 15:25:03 +00:00
Ryo Onodera 611af87fdb
Really start caching by fixing swapped CAS... (#18842) 2021-07-23 10:17:19 +09:00
Brooks Prumo d1debcd971
Add incremental snapshot utils (#18504) 
This commit adds high-level functions for creating and loading-from
incremental snapshots, plus all low-level functions required to perform
those tasks.  This commit **does not** add taking incremental snapshots
as part of a running validator, nor starting up a node with an
incremental snapshot; just laying ground work.

Additionally, `snapshot_utils` and `serde_snapshot` have been
refactored to use a common code paths for the different snapshots.

Also of note, some renaming has happened:
  1. Snapshots are now either `full_` or `incremental_` throughout the
     codebase.  If not specified, the code applies to both.
  2. Bank snapshots now are called "bank snapshots"
     (before they were called "slot snapshots", "bank snapshots", or
      just "snapshots").  The one exception is within `Bank`, where they
     are still just "snapshots", because they are already "bank
     snapshots".
  3. Snapshot archives now have `_archive` in the code.  This
     should clear up an ambiguity between bank snapshots and snapshot
     archives.
 2021-07-22 14:40:37 -05:00
behzad nouri 7d56fa8363
sends packets in batches from sigverify-stage (#18446) 
sigverify-stage is breaking batches to single-item vectors before
sending them down the channel:
https://github.com/solana-labs/solana/blob/d451363dc/core/src/sigverify_stage.rs#L88-L92

Also simplifying window-service code, reducing number of nested branches.
 2021-07-22 14:49:21 +00:00
Michael Vines 61865c0ee0 `solana-validator set-identity` now loads the tower file for the new identity 2021-07-21 22:22:08 -07:00
carllin 588c0464b8
Add sampling logic and DuplicateSlotRepairStatus module (#18721) 2021-07-21 11:15:08 -07:00
behzad nouri bbd22f06f4
implements generic lookups into gossip crds table (#18765) 
This commit adds CrdsEntry trait which allows generic lookups into crds
table. For example to get ContactInfo or LowestSlot associated with a
Pubkey, the lookup code would be respectively:
   crds.get::<&ContactInfo>(pubkey)
   crds.get::<&LowestSlot>(pubkey)
 2021-07-21 12:16:26 +00:00
carllin ce467bea20
Add frozen hashes and marking DuplicateConfirmed in blockstore to state machine (#18648) 2021-07-18 17:04:25 -07:00
behzad nouri e316586516 excludes private ip addresses 2021-07-16 20:05:48 -06:00
Jeff Biseda ae5ad5cf9b
sendmmsg cleanup #18589 
Rationalize usage of sendmmsg(2). Skip packets which failed to send and track failures.
 2021-07-16 14:36:49 -07:00
Jack May ca71ca3d6d
Accumulate consumed units (#18714) 2021-07-16 12:40:12 -07:00
Justin Starry d166b9856a
Move transaction sanitization earlier in the pipeline (#18655) 
* Move transaction sanitization earlier in the pipeline

* Renamed HashedTransaction to SanitizedTransaction

* Implement deref for sanitized transaction

* bring back process_transactions test method

* Use sanitized transactions for cost model calculation
 2021-07-15 22:51:27 -05:00
carllin 8a846b048e
Add AncestorHashesRepair type (#18681) 2021-07-15 19:29:53 -07:00
Trent Nelson 3a85b77bb5 hijack secp256k1 enablement feature plumbing for libsecp256k1 upgrade 2021-07-15 18:43:55 +00:00
Trent Nelson 568660b402 Revert "Remove feature switch for secp256k1 program (#18467)" 
This reverts commit fd574dcb3b.
 2021-07-15 18:43:55 +00:00
sakridge 0f8bcf65af
Add voting service (#18552) 2021-07-15 16:35:51 +02:00
behzad nouri cf31afdd6a
makes CrdsGossip thread-safe (#18615) 2021-07-14 22:27:17 +00:00
Michael Vines b30b32300d `solana-validator set-identity` now works for voting validators 2021-07-14 09:42:35 -07:00
Michael Vines 62d864559f Tower cleanup: reduce fn visibility, remove unnecessary new_with_key() 2021-07-14 09:42:35 -07:00
sakridge 7f2254225e
Move entry/poh to own crate to speed up poh bench build (#18225) 2021-07-14 14:16:29 +02:00
behzad nouri c90af3cd63
removes id from push_lowest_slot args (#18645) 
push_lowest_slot cannot sign the new crds-value unless the id (pubkey)
argument passed-in is the same pubkey as in ClusterInfo::keypair(), in
which case the id argument is redundant:
https://github.com/solana-labs/solana/blob/bb41cf346/gossip/src/cluster_info.rs#L824-L845

Additionally, the lookup is done with self.id(), but insert is done with
the id argument, which is logically a bug.
 2021-07-13 22:32:59 +00:00
Tao Zhu 350baece21
Explicitly sanitize program id indexes before usage 
1. check transaction has valid program_id before using it to avoid possible panic;
2. change calculate_cost function signature to return Result;
3. add CostModelError enum, update return type from Result<_, str> to Result<_, CostModelError>
 2021-07-13 17:29:22 -05:00
Jeff Washington (jwash) d092fa1f03
add ledger-tool verify verify-accounts-index option (#18375) 
* add ledger-tool verify verify-accounts-index option

* comment, merge, respond to feedback, cleanup
 2021-07-13 11:06:18 -05:00
Michael Vines 4098af3b5b Record vote account commission with voting/staking rewards and surface in RPC 2021-07-12 15:09:44 -07:00
carllin 175083c4c1
Add updated duplicate broadcast test (#18506) 2021-07-10 22:22:07 -07:00
Jack May e9ace3a0d5
cost model nits (#18528) 2021-07-09 12:55:31 -07:00
Justin Starry fd574dcb3b
Remove feature switch for secp256k1 program (#18467) 
* Remove feature switch for secp256k1 program

* fix tests
 2021-07-09 10:08:03 -05:00
carllin 4d3e301ee4
Introduce slot dumping to ReplayStage (#18160) 2021-07-08 19:07:32 -07:00
dependabot[bot] c853da7424
chore: bump libc from 0.2.97 to 0.2.98 (#18516) 
* chore: bump libc from 0.2.97 to 0.2.98

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.97 to 0.2.98.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.97...0.2.98)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-07-08 17:31:16 +00:00
Tao Zhu b6dff12923
update ledger tool to restore cost table from blockstore (#18489) 
* update ledger tool to restore cost model from blockstore when compute-slot-cost

* Move initialize_cost_table into cost_model, so the function can be tested and shared between validator and ledger-tool

* refactor and simplify a test
 2021-07-07 23:44:51 -05:00
Michael Vines 1e0942e900 Rename ClusterInfo::send_vote to ClusterInfo::send_transaction 2021-07-07 15:51:14 -07:00
jbiseda a86ced0bac
generate deterministic seeds for shreds (#17950) 
* generate shred seed from leader pubkey

* clippy

* clippy

* review

* review 2

* fmt

* review

* check

* review

* cleanup

* fmt
 2021-07-07 08:21:12 -07:00
behzad nouri a0551b4054
persists repair-peers cache across repair service loops (#18400) 
The repair-peers cache is reset each time repair service loop runs,
and so computed repeatedly for the same slots:
https://github.com/solana-labs/solana/blob/d2b07dca9/core/src/repair_service.rs#L275

This commit uses an LRU cache to persists repair-peers for each slot.
In addition to LRU eviction rules, in order to avoid re-using outdated
data, each entry also has 10 seconds TTL.
 2021-07-07 14:12:09 +00:00
behzad nouri 04787be8b1
encapsulates turbine peers computations of broadcast & retransmit stages (#18238) 
Broadcast stage and retransmit stage should arrange nodes on turbine
broadcast tree in exactly same order. Additionally any changes to this
ordering (e.g. updating how unstaked nodes are handled) requires feature
gating to keep the cluster in sync.

Current implementation is scattered out over several public methods and
exposes too much of implementation details (e.g. usize indices into
peers vector) which makes code changes and checking for feature
activations more difficult.

This commit encapsulates turbine peer computations into a new struct,
and only exposes two public methods, get_broadcast_peer and
get_retransmit_peers, for call-sites.
 2021-07-07 00:35:25 +00:00
Justin Starry 100fabf469
Remove feature switch for demoting sysvar write locks (#18373) 2021-07-06 21:22:22 +00:00
Tao Zhu 0e039b4094
Aggregate cost_model into cost_tracker (#18374) 
* * aggregate cost_model into cost_tracker, decouple it from banking_stage to prevent accidental deadlock. * Simplified code, removed unused functions

* review fixes
 2021-07-06 15:41:25 +00:00
Michael Vines d5c2c72360 Rename Tower::lockouts to Tower::vote_state 2021-07-02 18:35:49 -07:00
Tao Zhu 7cd6224caf
log warning when channel send fails (#18391) 2021-07-02 19:04:09 +00:00
carllin 0eca92de18
Make set roots an iterator (#18357) 2021-07-01 20:02:40 -07:00
Michael Vines b6792a3328 Add ability to change the validator identity at runtime 2021-07-01 17:50:04 -07:00
Brooks Prumo 45d54b1fc6
Add SnapshotArchiveInfo and refactor functions in snapshot_utils (#18232) 2021-07-01 12:20:56 -05:00
Tao Zhu 5e424826ba
Persist cost table to blockstore (#18123) 
* Add `ProgramCosts` Column Family to blockstore, implement LedgerColumn; add `delete_cf` to Rocks
* Add ProgramCosts to compaction excluding list alone side with TransactionStatusIndex in one place: `excludes_from_compaction()`

* Write cost table to blockstore after `replay_stage` replayed active banks; add stats to measure persist time
* Deletes program from `ProgramCosts` in blockstore when they are removed from cost_table in memory
* Only try to persist to blockstore when cost_table is changed.
* Restore cost table during validator startup

* Offload `cost_model` related operations from replay main thread to dedicated service thread, add channel to send execute_timings between these threads;
* Move `cost_update_service` to its own module; replay_stage is now decoupled from cost_model.
 2021-07-01 11:32:41 -05:00
dependabot[bot] 05924423c2
chore: bump reqwest from 0.11.2 to 0.11.4 (#18362) 
* chore: bump reqwest from 0.11.2 to 0.11.4

Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.2 to 0.11.4.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.2...v0.11.4)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-07-01 09:59:30 -06:00
Brooks Prumo 89a3e4f91e
Move SnapshotConfig into its own module (#18331) 
Also move ArchiveFormat to snapshot_utils, and do not
reexport SnapshotVersion.
 2021-07-01 08:55:26 -05:00
dependabot[bot] 78968d132f
chore: bump log from 0.4.11 to 0.4.14 (#18323) 
* chore: bump log from 0.4.11 to 0.4.14

Bumps [log](https://github.com/rust-lang/log) from 0.4.11 to 0.4.14.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/compare/0.4.11...0.4.14)

---
updated-dependencies:
- dependency-name: log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Make version consistent

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-06-30 18:32:01 +00:00
sakridge 8d9a6deda4
Add repair number per slot (#18082) 2021-06-30 18:20:07 +02:00
dependabot[bot] e9165232ef
chore: bump indexmap from 1.6.2 to 1.7.0 (#18322) 
* chore: bump indexmap from 1.6.2 to 1.7.0

Bumps [indexmap](https://github.com/bluss/indexmap) from 1.6.2 to 1.7.0.
- [Release notes](https://github.com/bluss/indexmap/releases)
- [Commits](https://github.com/bluss/indexmap/compare/1.6.2...1.7.0)

---
updated-dependencies:
- dependency-name: indexmap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-06-30 09:25:01 -06:00
Trent Nelson 02b14caa5f test-validator: hold rent constant with `--slots-per-epoch` 2021-06-30 00:46:12 -06:00
dependabot[bot] 52fd10ce03
chore: bump byteorder from 1.3.4 to 1.4.3 (#18300) 
* chore: bump byteorder from 1.3.4 to 1.4.3

Bumps [byteorder](https://github.com/BurntSushi/byteorder) from 1.3.4 to 1.4.3.
- [Release notes](https://github.com/BurntSushi/byteorder/releases)
- [Changelog](https://github.com/BurntSushi/byteorder/blob/master/CHANGELOG.md)
- [Commits](https://github.com/BurntSushi/byteorder/compare/1.3.4...1.4.3)

---
updated-dependencies:
- dependency-name: byteorder
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* Bump expected instruction count

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-06-29 15:08:04 -06:00
carllin 68c87469c3
Cleanup ReplayStage tests (#18241) 2021-06-28 20:19:42 -07:00
Tao Zhu 9d6f1ebef4
investigate system performance test degradation (#17919) 
* Add stats and counter around cost model ops, mainly:
- calculate transaction cost
- check transaction can fit in a block
- update block cost tracker after transactions are added to block
- replay_stage to update/insert execution cost to table

* Change mutex on cost_tracker to RwLock

* removed cloning cost_tracker for local use, as the metrics show clone is very expensive.

* acquire and hold locks for block of TXs, instead of acquire and release per transaction;

* remove redundant would_fit check from cost_tracker update execution path

* refactor cost checking with less frequent lock acquiring

* avoid many Transaction_cost heap allocation when calculate cost, which
is in the hot path - executed per transaction.

* create hashmap with new_capacity to reduce runtime heap realloc.

* code review changes: categorize stats, replace explicit drop calls, concisely initiate to default

* address potential deadlock by acquiring locks one at time
 2021-06-28 21:34:04 -05:00
dependabot[bot] 7782a0855d
chore: bump rand_core from 0.6.2 to 0.6.3 (#18263) 
* chore: bump rand_core from 0.6.2 to 0.6.3

Bumps [rand_core](https://github.com/rust-random/rand) from 0.6.2 to 0.6.3.
- [Release notes](https://github.com/rust-random/rand/releases)
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/rand/compare/rand_core-0.6.2...rand_core-0.6.3)

---
updated-dependencies:
- dependency-name: rand_core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-06-28 12:07:27 -06:00
sakridge 5d08bf9aa3
More detailed voting timings in replay stage (#18229) 2021-06-26 17:32:08 +02:00
dependabot[bot] 91576fdd9b
chore: bump blake3 from 0.3.7 to 0.3.8 (#18223) 
* chore: bump blake3 from 0.3.7 to 0.3.8

Bumps [blake3](https://github.com/BLAKE3-team/BLAKE3) from 0.3.7 to 0.3.8.
- [Release notes](https://github.com/BLAKE3-team/BLAKE3/releases)
- [Commits](https://github.com/BLAKE3-team/BLAKE3/compare/0.3.7...0.3.8)

---
updated-dependencies:
- dependency-name: blake3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
 2021-06-25 11:44:52 -06:00
dependabot[bot] 767c6eb57d
chore: bump miow from 0.3.6 to 0.3.7 (#18222) 
* chore: bump miow from 0.3.6 to 0.3.7

Bumps [miow](https://github.com/yoshuawuyts/miow) from 0.3.6 to 0.3.7.
- [Release notes](https://github.com/yoshuawuyts/miow/releases)
- [Changelog](https://github.com/yoshuawuyts/miow/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yoshuawuyts/miow/compare/v0.3.6...v0.3.7)

---
updated-dependencies:
- dependency-name: miow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-06-25 11:08:57 -06:00
Trent Nelson d269975784 Revert "Clean up build warning" 
This reverts commit 17a173ebb5.
 2021-06-24 19:57:52 -06:00
Michael Vines 314102cb54 Remove redundant JsonRpcConfig::identity_pubkey field 2021-06-22 17:20:11 -07:00
dependabot[bot] 6fc73470df
chore: bump raptorq from 1.4.2 to 1.6.3 (#18137) 
Bumps [raptorq](https://github.com/cberner/raptorq) from 1.4.2 to 1.6.3.
- [Release notes](https://github.com/cberner/raptorq/releases)
- [Commits](https://github.com/cberner/raptorq/compare/v1.4.2...v1.6.3)

---
updated-dependencies:
- dependency-name: raptorq
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-22 17:28:11 +00:00
dependabot[bot] 2156712768
chore: bump lru from 0.6.1 to 0.6.5 (#18138) 
Bumps [lru](https://github.com/jeromefroe/lru-rs) from 0.6.1 to 0.6.5.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases)
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jeromefroe/lru-rs/compare/0.6.1...0.6.5)

---
updated-dependencies:
- dependency-name: lru
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-22 17:07:45 +00:00
sakridge e808f34b0b
Add batch stats (#18096) 2021-06-22 15:23:26 +02:00
Michael Vines 3b1517237c Clean up argument names 2021-06-21 21:29:52 -07:00
Michael Vines 84b9de8c18 Shredder no longer holds a keypair 2021-06-21 21:29:52 -07:00
Michael Vines 2435ea3ad8 Remove redundant ReplayStageConfig::my_pubkey field 2021-06-21 21:29:52 -07:00
Michael Vines 51a0007001 serve_repair: Remove internal ContactInfo field duplication 2021-06-21 17:23:49 -07:00
behzad nouri 598093b5db adds shred-version to ip-echo-server response 
When starting a validator, the node initially joins gossip with
shred_verison = 0, until it adopts the entrypoint's shred-version:
https://github.com/solana-labs/solana/blob/9b182f408/validator/src/main.rs#L417

Depending on the load on the entrypoint, this adopting entrypoint
shred-version through gossip sometimes becomes very slow, and causes
several problems in gossip because we have to partially support
shred_version == 0 which is a source of leaking crds values from one
cluster to another. e.g. see
https://github.com/solana-labs/solana/pull/17899
and the other linked issues there.

In order to remove shred_version == 0 from gossip, this commit adds
shred-version to ip-echo-server response. Once the entrypoints are
updated, on validator start-up, if --expected_shred_version is not
specified we will obtain shred-version from the entrypoint using
ip-echo-server.
 2021-06-21 19:37:16 +00:00
dependabot[bot] d458fac2ff
chore: bump bincode from 1.3.1 to 1.3.3 (#18087) 
* chore: bump bincode from 1.3.1 to 1.3.3

Bumps [bincode](https://github.com/servo/bincode) from 1.3.1 to 1.3.3.
- [Release notes](https://github.com/servo/bincode/releases)
- [Commits](https://github.com/servo/bincode/compare/v1.3.1...v1.3.3)

---
updated-dependencies:
- dependency-name: bincode
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-06-21 16:06:31 +00:00
Jeff Washington (jwash) ec2f930475
user process.accounts_db_test_hash_calculation for debug_verify hash (#18053) 2021-06-21 10:20:27 -05:00
Michael Vines 4a12c715a3 Drop Error suffix from enum values to avoid the enum_variant_names clippy lint 2021-06-18 23:02:13 +00:00
dependabot[bot] 491e4450cc
chore: bump libc from 0.2.96 to 0.2.97 (#18060) 
* chore: bump libc from 0.2.96 to 0.2.97

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.96 to 0.2.97.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.96...0.2.97)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-06-18 11:51:06 -06:00
Alexander Meißner 789f33e8db chore: cargo fmt 2021-06-18 10:42:46 -07:00
Alexander Meißner 6514096a67 chore: cargo +nightly clippy --fix -Z unstable-options 2021-06-18 10:42:46 -07:00
Tyera Eulberg d0511de9a6
chore: bump trees from 0.2.1 to 0.4.2 (#18052) 
* chore: bump trees from 0.2.1 to 0.4.2 (#18041)

Bumps [trees](https://github.com/oooutlk/trees) from 0.2.1 to 0.4.2.
- [Release notes](https://github.com/oooutlk/trees/releases)
- [Commits](https://github.com/oooutlk/trees/commits)

---
updated-dependencies:
- dependency-name: trees
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Accommodate field & type changes

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-17 22:45:09 +00:00
Jeff Washington (jwash) bab96e878e
Revert "chore: bump trees from 0.2.1 to 0.4.2 (#18041)" (#18045) 
This reverts commit f2a7152c0c.
 2021-06-17 14:04:59 -05:00
Lijun Wang 071b1ee3e5
Removed pub from some functions which are actually private to improve encapsulation (#18030) 
Remove the pub marker to improve encapsulation. Readability improvement only, no functional impact.
 2021-06-17 10:14:21 -07:00
dependabot[bot] f2a7152c0c
chore: bump trees from 0.2.1 to 0.4.2 (#18041) 
Bumps [trees](https://github.com/oooutlk/trees) from 0.2.1 to 0.4.2.
- [Release notes](https://github.com/oooutlk/trees/releases)
- [Commits](https://github.com/oooutlk/trees/commits)

---
updated-dependencies:
- dependency-name: trees
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-17 08:07:16 -06:00
Michael Vines fa04531c7a Extricate RpcCompletedSlotsService from RetransmitStage 2021-06-16 16:20:35 -07:00
Trent Nelson 5bc6c89adc validator: run poh speed test earlier in start up 2021-06-16 21:27:08 +00:00
dependabot[bot] 6f5b248746
chore: bump miow from 0.2.2 to 0.3.6 (#18000) 
* chore: bump miow from 0.2.2 to 0.3.6

Bumps [miow](https://github.com/yoshuawuyts/miow) from 0.2.2 to 0.3.6.
- [Release notes](https://github.com/yoshuawuyts/miow/releases)
- [Changelog](https://github.com/yoshuawuyts/miow/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yoshuawuyts/miow/compare/0.2.2...v0.3.6)

---
updated-dependencies:
- dependency-name: miow
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-06-16 12:30:17 -06:00
Jeff Washington (jwash) dbd4dc04b0
ledger tool limit_load_slot_count_from_snapshot avoids assert failures (#17974) 2021-06-15 15:39:22 -05:00
Jeff Washington (jwash) f558b9b6bf
verify bank hash on startup with ledger tool option (#17939) 2021-06-15 11:52:12 -05:00
Jon Cinque 1b1d34da59
Refactor stake program into solana_program (#17906) 
* Move stake state / instructions into solana_program

* Update account-decoder

* Update cli and runtime

* Update all other parts

* Commit Cargo.lock changes in programs/bpf

* Update cli stake instruction import

* Allow integer arithmetic

* Update ABI digest

* Bump rust mem instruction count

* Remove useless structs

* Move stake::id() -> stake::program::id()

* Re-export from solana_sdk and mark deprecated

* Address feedback

* Run cargo fmt
 2021-06-15 18:04:00 +02:00
behzad nouri 161838655c
removes port-based forwarding logic from turbine retransmit (#17716) 
Turbine retransmit logic is based on which socket it received the packet
from (i.e `packet.meta.forward`):
https://github.com/solana-labs/solana/blob/708bbcb00/core/src/retransmit_stage.rs#L467-L470

This can leave the cluster vulnerable to spoofing and selective
propagation of packets; see
https://github.com/solana-labs/solana/issues/6672
https://github.com/solana-labs/solana/pull/7774

This commit identifies if the node is on the "critical path" based on
its index in the shuffled cluster. If so, it forwards the packet to both
neighbors and children; otherwise, the packet is only forwarded to the
children.

The metrics added in
https://github.com/solana-labs/solana/pull/17351
shows that the number of times the index does not match the port is very
rare, and therefore this change should be safe.
 2021-06-15 13:19:41 +00:00
carllin ccc013e134
Handle removing slots during account scans (#17471) 2021-06-14 21:04:01 -07:00
Jeff Washington (jwash) 471b34132e
add metrics for startup (#17913) 
* add metrics for startup

* roll timings up higher

* fix test

* fix duplicate
 2021-06-14 17:46:49 -05:00
Jeff Washington (jwash) e6bbd4b3f0
add metrics to handle_snapshot_requests (#17937) 2021-06-14 15:46:19 -05:00
dependabot[bot] a0872232d3
chore: bump itertools from 0.9.0 to 0.10.1 (#17929) 
* chore: bump itertools from 0.9.0 to 0.10.1

Bumps [itertools](https://github.com/rust-itertools/itertools) from 0.9.0 to 0.10.1.
- [Release notes](https://github.com/rust-itertools/itertools/releases)
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.9.0...v0.10.1)

---
updated-dependencies:
- dependency-name: itertools
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix versions

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-06-14 18:32:20 +00:00
dependabot[bot] a248770a6b
chore: bump systemstat from 0.1.7 to 0.1.8 (#17928) 
Bumps [systemstat](https://github.com/unrelentingtech/systemstat) from 0.1.7 to 0.1.8.
- [Release notes](https://github.com/unrelentingtech/systemstat/releases)
- [Commits](https://github.com/unrelentingtech/systemstat/compare/v0.1.7...v0.1.8)

---
updated-dependencies:
- dependency-name: systemstat
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-14 16:48:54 +00:00
sakridge eeee75c5be
Don't use pinned memory when unnecessary (#17832) 
Reports of excessive GPU memory usage and errors
from cudaHostRegister. There are some cases where pinning is
not required.
 2021-06-14 16:10:04 +02:00
dependabot[bot] a501707058
chore: bump ahash from 0.6.1 to 0.7.4 (#17886) 
Bumps [ahash](https://github.com/tkaitchuck/ahash) from 0.6.1 to 0.7.4.
- [Release notes](https://github.com/tkaitchuck/ahash/releases)
- [Commits](https://github.com/tkaitchuck/ahash/commits)

---
updated-dependencies:
- dependency-name: ahash
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-11 14:25:25 -06:00
sakridge 0feac57cb0
Don't store votes unless we are leader soon (#17803) 2021-06-11 18:29:05 +02:00
dependabot[bot] 2aa7df23b5
chore: bump indexmap from 1.5.1 to 1.6.2 (#17884) 
* chore: bump indexmap from 1.5.1 to 1.6.2

Bumps [indexmap](https://github.com/bluss/indexmap) from 1.5.1 to 1.6.2.
- [Release notes](https://github.com/bluss/indexmap/releases)
- [Commits](https://github.com/bluss/indexmap/compare/1.5.1...1.6.2)

---
updated-dependencies:
- dependency-name: indexmap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-06-11 09:50:09 -06:00
carllin c8535be0e1
Port unconfirmed duplicate tracking logic from ProgressMap to ForkChoice (#17779) 2021-06-11 03:09:57 -07:00
carllin afafa624a3
Account for duplicate before a bank is frozen or replayed (#17866) 2021-06-10 22:28:23 -07:00
dependabot[bot] f08ed1eb2d
chore: bump rayon from 1.5.0 to 1.5.1 (#17869) 
* chore: bump rayon from 1.5.0 to 1.5.1

Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.5.0 to 1.5.1.
- [Release notes](https://github.com/rayon-rs/rayon/releases)
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md)
- [Commits](https://github.com/rayon-rs/rayon/compare/rayon-core-v1.5.0...v1.5.1)

---
updated-dependencies:
- dependency-name: rayon
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-06-10 13:18:47 -06:00
Lijun Wang 269d995832
Make account shrink configurable #17544 (#17778) 
1. Added both options for measuring space usage using total accounts usage and for individual store shrink ratio using an enum. Validator CLI options: --accounts-shrink-optimize-total-space and --accounts-shrink-ratio
2. Added code for selecting candidates based on total usage in a separate function select_candidates_by_total_usage
3. Added unit tests for the new functions added
4. The default implementations is kept at 0.8 shrink ratio with --accounts-shrink-optimize-total-space set to true

Fixes #17544
 2021-06-09 21:21:32 -07:00
Tao Zhu ae27fcbcda
replay stage feed back program cost (#17731) 
* replay stage feeds back realtime per-program execution cost to cost model;

* program cost execution table is initialized into empty table, no longer populated with hardcoded numbers;

* changed cost unit to microsecond, using value collected from mainnet;

* add ExecuteCostTable with fixed capacity for security concern, when its limit is reached, programs with old age AND less occurrence will be pushed out to make room for new programs.
 2021-06-09 17:10:59 -05:00
Justin Starry 050bb5446d
Add local cluster tests that broadcast duplicate slots (#13995) 
* Add duplicate node local cluster test

* fix clippy

* remove dupe test
 2021-06-09 15:01:48 -07:00
dependabot[bot] 9a2ca8dd2f
chore: bump rustc_version from 0.2.3 to 0.4.0 (#17854) 
* chore: bump rustc_version from 0.2.3 to 0.4.0

Bumps [rustc_version](https://github.com/Kimundi/rustc-version-rs) from 0.2.3 to 0.4.0.
- [Release notes](https://github.com/Kimundi/rustc-version-rs/releases)
- [Commits](https://github.com/Kimundi/rustc-version-rs/compare/v0.2.3...v0.4.0)

---
updated-dependencies:
- dependency-name: rustc_version
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Make versions consistent

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-06-09 16:57:39 +00:00
Michael Vines e5e7390d44 Wrap long lines 2021-06-08 12:05:29 -07:00
Tyera Eulberg 2e998ed11d
Bump jsonrpc crates (#17797) 
* Bump jsonrpc crates

* Update error text

* gitignore for rpc
 2021-06-07 18:03:20 +00:00
dependabot[bot] 8a77b21be1
chore: bump retain_mut from 0.1.2 to 0.1.3 (#17736) 
Bumps [retain_mut](https://github.com/upsuper/retain_mut) from 0.1.2 to 0.1.3.
- [Release notes](https://github.com/upsuper/retain_mut/releases)
- [Commits](https://github.com/upsuper/retain_mut/compare/v0.1.2...v0.1.3)

---
updated-dependencies:
- dependency-name: retain_mut
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-04 17:36:21 +00:00
Tyera Eulberg 544b3c0d17
Create solana-poh and move remaining rpc modules to solana-rpc (#17698) 
* Create solana-poh crate

* Move BigTableUploadService to solana-ledger

* Add solana-rpc to workspace

* Move dependencies to solana-rpc

* Move remaining rpc modules to solana-rpc

* Single use statement solana-poh

* Single use statement solana-rpc
 2021-06-04 09:23:06 -06:00
sakridge f97ce2cd7e
Per-program id timings (#17554) 2021-06-04 16:04:31 +02:00
behzad nouri be957f25c9
adds fallback logic if retransmit multicast fails (#17714) 
In retransmit-stage, based on the packet.meta.seed and resulting
children/neighbors, each packet is sent to a different set of peers:
https://github.com/solana-labs/solana/blob/708bbcb00/core/src/retransmit_stage.rs#L421-L457

However, current code errors out as soon as a multicast call fails,
which will skip all the remaining packets:
https://github.com/solana-labs/solana/blob/708bbcb00/core/src/retransmit_stage.rs#L467-L470

This can exacerbate packets loss in turbine.

This commit:
  * keeps iterating over retransmit packets for loop even if some
    intermediate sends fail.
  * adds a fallback to UdpSocket::send_to if multicast fails.

Recent discord chat:
https://discord.com/channels/428295358100013066/689412830075551748/849530845052403733
 2021-06-04 12:16:37 +00:00
Tyera Eulberg 3a647c4bea
Rename ValidatorExit and move to sdk (#17728) 2021-06-04 03:06:13 +00:00
dependabot[bot] 3670435db4
chore: bump serial_test from 0.4.0 to 0.5.1 (#17705) 
Bumps [serial_test](https://github.com/palfrey/serial_test) from 0.4.0 to 0.5.1.
- [Release notes](https://github.com/palfrey/serial_test/releases)
- [Commits](https://github.com/palfrey/serial_test/compare/v0.4.0...v0.5.1)

---
updated-dependencies:
- dependency-name: serial_test
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-03 10:07:56 -06:00
carllin 96ba2edfeb
Switch EpochSlots to be frozen slots, not completed slots (#17168) 2021-06-03 00:20:00 +00:00
dependabot[bot] 19c49e2e25
chore: bump crossbeam-channel from 0.4.4 to 0.5.0 (#17665) 
* chore: bump crossbeam-channel from 0.4.4 to 0.5.0

Bumps [crossbeam-channel](https://github.com/crossbeam-rs/crossbeam) from 0.4.4 to 0.5.0.
- [Release notes](https://github.com/crossbeam-rs/crossbeam/releases)
- [Changelog](https://github.com/crossbeam-rs/crossbeam/blob/master/CHANGELOG.md)
- [Commits](https://github.com/crossbeam-rs/crossbeam/compare/crossbeam-channel-0.4.4...crossbeam-channel-0.5.0)

---
updated-dependencies:
- dependency-name: crossbeam-channel
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-06-02 19:36:28 +00:00
dependabot[bot] d99c888cc2
chore: bump systemstat from 0.1.5 to 0.1.7 (#17667) 
Bumps [systemstat](https://github.com/myfreeweb/systemstat) from 0.1.5 to 0.1.7.
- [Release notes](https://github.com/myfreeweb/systemstat/releases)
- [Commits](https://github.com/myfreeweb/systemstat/commits/v0.1.7)

---
updated-dependencies:
- dependency-name: systemstat
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-02 09:08:07 -06:00
carllin bbcdf073ba
Support out of band dumping of unrooted slots in AccountsDb (#17269) 
* Accounts dumping logic

* Add test for interaction between cache flush and remove_unrooted_slot()

* Update comments

* Rename

* renaming

* Add more comments

* Renaming

* Fixup test and bad check
 2021-06-02 09:51:10 +00:00
Tao Zhu b000d490ce
Cost Model to limit transactions which are not parallelizeable (#16694) 
* * Add following to banking_stage:
  1. CostModel as immutable ref shared between threads, to provide estimated cost for transactions.
  2. CostTracker which is shared between threads, tracks transaction costs for each block.

* replace hard coded program ID with id() calls

* Add Account Access Cost as part of TransactionCost. Account Access cost are weighted differently between read and write, signed and non-signed.

* Establish instruction_execution_cost_table, add function to update or insert instruction cost, unit tested. It is read-only for now; it allows Replay to insert realtime instruction execution costs to the table.

* add test for cost_tracker atomically try_add operation, serves as safety guard for future changes

* check cost against local copy of cost_tracker, return transactions that would exceed limit as unprocessed transaction to be buffered; only apply bank processed transactions cost to tracker;

* bencher to new banking_stage with max cost limit to allow cost model being hit consistently during bench iterations
 2021-06-01 09:16:17 -05:00
dependabot[bot] ab0f4ff835
Bump serde from 1.0.122 to 1.0.126 (#17618) 
* Bump serde from 1.0.122 to 1.0.126

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.122 to 1.0.126.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.122...v1.0.126)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
 2021-05-31 22:41:25 +00:00
dependabot[bot] ac452f9bb8
Bump tempfile from 3.1.0 to 3.2.0 (#17587) 
* Bump tempfile from 3.1.0 to 3.2.0

Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.1.0 to 3.2.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases)
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.1.0...v3.2.0)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-05-31 18:05:07 +00:00
dependabot[bot] 86c07bf785
Bump bs58 from 0.3.1 to 0.4.0 (#17573) 
* Bump bs58 from 0.3.1 to 0.4.0

Bumps [bs58](https://github.com/mycorrhiza/bs58-rs) from 0.3.1 to 0.4.0.
- [Release notes](https://github.com/mycorrhiza/bs58-rs/releases)
- [Commits](https://github.com/mycorrhiza/bs58-rs/compare/0.3.1...0.4.0)

Signed-off-by: dependabot[bot] <support@github.com>

* Fixup

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-05-28 18:30:34 +00:00
Ryo Onodera 1f97b2365f
Avoid full-range compactions with periodic filtered b.g. ones (#16697) 
* Update rocksdb to v0.16.0

* Promote the infrequent and important log to info!

* Force background compaction by ttl without manual compaction

* Fix test

* Support no compaction mode in test_ledger_cleanup_compaction

* Fix comment

* Make compaction_interval customizable

* Avoid major compaction with periodic filtering...

* Adress lazy_static, special cfs and range check

* Clean up a bit and add comment

* Add comment

* More comments...

* Config code cleanup

* Add comment

* Use .conflicts_with()

* Nullify unneeded delete_range ops for special CFs

* Some clean ups

* Clarify the locking intention

* Ensure special CFs' consistency with PurgeType::CompactionFilter

* Fix comment

* Fix bad copy paste

* Fix various types...

* Don't use tuples

* Add a unit test for compaction_filter

* Fix typo...

* Remove flag and just use new behavior always

* Fix wrong condition negation...

* Doc. about no set_last_purged_slot in purge_slots

* Write a test and fix off-by-one bug....

* Apply suggestions from code review

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* Follow up to github review suggestions

* Fix line-wrapping

* Fix conflict

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
 2021-05-28 16:42:56 +09:00
Michael Vines 8eab0e8602
Bump version to v1.8.0 (#17541) 2021-05-27 08:51:53 -07:00
Tyera Eulberg ab581dafc2
Add block height to ConfirmedBlock structs (#17523) 
* Add BlockHeight CF to blockstore

* Rename CacheBlockTimeService to be more general

* Cache block-height using service

* Fixup previous proto mishandling

* Add block_height to block structs

* Add block-height to solana block

* Fallback to BankForks if block time or block height are not yet written to Blockstore

* Add docs

* Review comments
 2021-05-26 22:16:16 -06:00
Michael Vines 9541411c15 Plumb transaction-level rewards (aka "rent debits") into the `getTransaction` RPC method 2021-05-27 03:05:05 +00:00
carllin 52dccc656a
Purge slots greater than new last index (#16071) 2021-05-26 16:12:57 -07:00
Michael Vines cbce440af4 simulateTransaction can now return accounts modified by the simulation 2021-05-26 14:20:23 -07:00
Tyera Eulberg 6abe089740
Add custom error for tx-history queries when node does not support (#17494) 2021-05-26 13:27:41 -06:00
Jeff Washington (jwash) 6b9d8d41a3
add --limit_load_slot_count_from_snapshot to ledger-tool (#17417) 2021-05-26 10:36:12 -05:00
Tyera Eulberg 9a5330b7eb
Move gossip modules into solana-gossip crate (#17352) 
* Move gossip modules to solana-gossip

* Update Protocol abi digest due to move

* Move gossip benches and hook up CI

* Remove unneeded Result entries

* Single use statements
 2021-05-26 09:15:46 -06:00
Tyera Eulberg e9bc1c6b07
Add last valid block height to rpc Fees (#17506) 
* Add last_valid_block_height to fees rpc

* Add getBlockHeight rpc

* Update docs
 2021-05-26 07:26:19 +00:00