0028442e14
rework defaults and construction of accountsdb, accounts, bank ( #19083 )
...
* rework defaults and construction of accountsdb, accounts, bank
* merge issues
2021-08-05 17:27:13 -05:00
e4be00fece
falls back on working-bank if root-bank::epoch-staked-nodes is none
...
bank.get_leader_schedule_epoch(shred_slot)
is one epoch after epoch_schedule.get_epoch(shred_slot).
At epoch boundaries, shred is already one epoch after the root-slot. So
we need epoch-stakes 2 epochs ahead of the root. But the root bank only
has epoch-stakes for one epoch ahead, and as a result looking up epoch
staked-nodes from the root-bank fails.
To be backward compatible with the current master code, this commit
implements a fallback on working-bank if epoch staked-nodes obtained
from the root-bank is none.
2021-08-05 21:47:33 +00:00
eaf927cf49
allows only one thread to update cluster-nodes cache entry for an epoch
...
If two threads simultaneously call into ClusterNodesCache::get for the
same epoch, and the cache entry is outdated, then both threads recompute
cluster-nodes for the epoch and redundantly overwrite each other.
This commit wraps ClusterNodesCache entries in Arc<Mutex<...>>, so that
when needed only one thread does the computations to update the entry.
2021-08-05 21:47:33 +00:00
fb69f45f14
adds fallback & metric for when epoch staked-nodes are none
2021-08-05 21:47:33 +00:00
50d0e830c9
unifies cluster-nodes computation & caching across turbine stages
...
Broadcast-stage is using epoch_staked_nodes based on the same slot that
shreds belong to:
https://github.com/solana-labs/solana/blob/049fb0417/core/src/broadcast_stage/standard_broadcast_run.rs#L208-L228
https://github.com/solana-labs/solana/blob/0cf52e206/core/src/broadcast_stage.rs#L342-L349
But retransmit-stage is using bank-epoch of the working-bank:
https://github.com/solana-labs/solana/blob/19bd30262/core/src/retransmit_stage.rs#L272-L289
So the two are not consistent at epoch boundaries where some nodes may
have a working bank (or similarly a root bank) lagging other nodes. As a
result the node which obtains a packet may construct turbine broadcast
tree inconsistently with its parent node in the tree and so some packets
may fail to reach all nodes in the tree.
2021-08-05 21:47:33 +00:00
aa32738dd5
uses cluster-nodes cache in broadcast-stage
...
* Current caching mechanism does not update cluster-nodes when the epoch
(and so epoch staked nodes) changes:
https://github.com/solana-labs/solana/blob/19bd30262/core/src/broadcast_stage/standard_broadcast_run.rs#L332-L344
* Additionally, the cache update has a concurrency bug in which the
thread which does compare_and_swap may be blocked when it tries to
obtain the write-lock on cache, while other threads will keep running
ahead with the outdated cache (since the atomic timestamp is already
updated).
In the new ClusterNodesCache, entries are keyed by epoch, and so if
epoch changes cluster-nodes will be recalculated. The time-to-live
eviction policy is also encapsulated and rigidly enforced.
2021-08-05 21:47:33 +00:00
30bec3921e
uses cluster-nodes cache in retransmit stage
...
The new cluster-nodes cache will:
* ensure cluster-nodes are recalculated if the epoch (and so the epoch
staked nodes) changes.
* encapsulate time-to-live eviction policy.
2021-08-05 21:47:33 +00:00
ecc1c7957f
implements cluster-nodes cache
...
Cluster nodes are cached keyed by the respective epoch from which stakes
are obtained, and so if epoch changes cluster-nodes will be recomputed.
A time-to-live eviction policy is enforced to refresh entries in case
gossip contact-infos are updated.
2021-08-05 21:47:33 +00:00
44b11154ca
sends slots (instead of stakes) through broadcast flow
...
Current broadcast code is computing stakes for each slot before sending
them down the channel:
https://github.com/solana-labs/solana/blob/049fb0417/core/src/broadcast_stage/standard_broadcast_run.rs#L208-L228
https://github.com/solana-labs/solana/blob/0cf52e206/core/src/broadcast_stage.rs#L342-L349
Since the stakes are a function of epoch the slot belongs to (and so
does not necessarily change from one slot to another), forwarding the
slot itself would allow better caching downstream.
In addition we need to invalidate the cache if the epoch changes (which
the current code does not do), and that requires to know which slot (and
so epoch) current broadcasted shreds belong to:
https://github.com/solana-labs/solana/blob/19bd30262/core/src/broadcast_stage/standard_broadcast_run.rs#L332-L344
2021-08-05 21:47:33 +00:00
b67ffab370
Add more API documentation for Rust RpcClient ( #19021 )
...
* Add doc links to Transaction API docs
* Add more RpcClient API docs
* Reflow some rpc_client docs
* Update client/src/rpc_client.rs
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Update client/src/rpc_client.rs
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Update client/src/rpc_client.rs
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Update sdk/src/transaction.rs
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Update RpcClient docs per review
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-08-05 15:03:33 -06:00
e368f10973
add _for_tests to new_no_wallclock_throttle ( #19086 )
2021-08-05 14:50:25 -05:00
a9014ceceb
Bank::default_for_tests() ( #19084 )
2021-08-05 11:53:29 -05:00
24207a09ac
remove AccountsIndex::default ( #19082 )
...
* accounts_db calls AccountsDb::new(bins)
* remove AccountsIndex::default
2021-08-05 11:38:53 -05:00
5cf28689e6
accounts_db calls AccountsDb::new(bins) ( #19068 )
2021-08-05 11:15:26 -05:00
40914de811
updates cluster-slots with root-bank instead of root-slot + bank-forks
...
ClusterSlots::update is taking both root-slot and bank-forks only to
later lookup root-bank from bank-forks, which is redundant. Also
potentially by the time bank-forks is locked to obtain root-bank,
root-slot may have already changed and so be inconsistent with the
root-slot passed in as the argument.
https://github.com/solana-labs/solana/blob/6d95d679c/core/src/cluster_slots.rs#L32-L39
https://github.com/solana-labs/solana/blob/6d95d679c/core/src/cluster_slots.rs#L122
2021-08-05 14:43:06 +00:00
2fc112edcf
removes unused code from cluster-slots
2021-08-05 14:43:06 +00:00
67788ad206
move AccountsIndex upsert into static WriteAccountMapEntry ( #18899 )
...
* rework accounts index to push upsert deeper
* clean up return value of upsert_existing_key
* upsert_existing_key -> update_key_if_exists
* upsert_new_key -> upsert
* upsert_item -> lock_and_update_slot_list
* update_static -> update_slot_list
2021-08-05 08:45:08 -05:00
bf16b0517c
add _for_tests to setup_bank_and_vote_pubkeys ( #19060 )
2021-08-05 08:43:35 -05:00
087db70df6
add traits required by IsCached ( #19066 )
2021-08-05 08:43:00 -05:00
14361906ca
for all tests, bank::new -> bank::new_for_tests ( #19064 )
2021-08-05 08:42:38 -05:00
367d5f62ce
chore:(deps): bump @solana/spl-token-registry in /explorer ( #19077 )
...
Bumps [@solana/spl-token-registry](https://github.com/solana-labs/token-list ) from 0.2.208 to 0.2.210.
- [Release notes](https://github.com/solana-labs/token-list/releases )
- [Changelog](https://github.com/solana-labs/token-list/blob/main/CHANGELOG.md )
- [Commits](https://github.com/solana-labs/token-list/compare/v0.2.208...v0.2.210 )
---
updated-dependencies:
- dependency-name: "@solana/spl-token-registry"
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-05 08:34:30 +00:00
6e285c42ea
chore:(deps): bump @types/node from 16.4.10 to 16.4.12 in /explorer ( #19076 )
...
Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node ) from 16.4.10 to 16.4.12.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases )
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node )
---
updated-dependencies:
- dependency-name: "@types/node"
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-05 08:30:56 +00:00
80e770dfa0
chore: bump @babel/preset-env from 7.14.9 to 7.15.0 in /web3.js ( #19073 )
...
Bumps [@babel/preset-env](https://github.com/babel/babel/tree/HEAD/packages/babel-preset-env ) from 7.14.9 to 7.15.0.
- [Release notes](https://github.com/babel/babel/releases )
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md )
- [Commits](https://github.com/babel/babel/commits/v7.15.0/packages/babel-preset-env )
---
updated-dependencies:
- dependency-name: "@babel/preset-env"
dependency-type: direct:development
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-05 08:16:48 +00:00
dfe99efa7c
introduce AccountsIndex::default_for_tests() ( #19067 )
2021-08-04 21:58:53 -05:00
5a4979f25f
Handle 0-lamport account in index generation ( #19041 )
...
* Handle 0-lamport account in index generation
* rename duplicate to dirty keys
Co-authored-by: Carl Lin <carl@solana.com>
2021-08-04 23:33:47 +00:00
bde9b4de94
Bank::new -> Bank::new_for_benches ( #19063 )
2021-08-04 17:30:43 -05:00
3280ae3e9f
add validator option --accounts-db-skip-shrink ( #19028 )
...
* add validator option --accounts-db-skip-shrink
* typo
2021-08-04 17:28:33 -05:00
68cc71409e
Do not shell out for tar ( #19043 )
...
When making a snapshot archive, we used to shell out and call `tar -S`
for sparse file support. The tar crate supports sparse files, so no
need to do this anymore.
Fixes #10860
2021-08-04 17:07:55 -05:00
a1112254a5
Fix wrong old snapshot archives getting purged ( #19061 )
...
I introduced a bug where old snapshot archives were incorrectly purged.
Instead of purged to oldest, I was purged the newest...
The fix is to add a `reverse()` in the purge logic, and I've added a
test to catch this bug in the future.
Fixes #19057
2021-08-04 16:42:42 -05:00
0b8d14b0fc
move towards account index being dynamically allocated ( #19034 )
2021-08-04 15:28:35 -05:00
1ed12a07ab
introduce Bank::new_for_tests ( #19062 )
2021-08-04 15:06:57 -05:00
ca14475085
Add incremental_snapshot_archive_interval_slots to SnapshotConfig ( #19026 )
...
This commit also renames `snapshot_interval_slots` to
`full_snapshot_archive_interval_slots`, updates the comments on the
fields, and make appropriate updates where SnapshotConfig is used.
2021-08-04 14:40:20 -05:00
cc27b8a5a7
chore: bump const_format from 0.2.15 to 0.2.17 ( #19049 )
...
Bumps [const_format](https://github.com/rodrimati1992/const_format_crates ) from 0.2.15 to 0.2.17.
- [Release notes](https://github.com/rodrimati1992/const_format_crates/releases )
- [Changelog](https://github.com/rodrimati1992/const_format_crates/blob/master/Changelog.md )
- [Commits](https://github.com/rodrimati1992/const_format_crates/commits )
---
updated-dependencies:
- dependency-name: const_format
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-04 13:38:50 -06:00
d60ccf64e1
bump recommended maps/nofiles
2021-08-04 11:15:18 -06:00
6a995f5dfd
rename to AccountsDb::new_single_for_tests ( #19039 )
2021-08-04 11:47:11 -05:00
6d95d679c4
chore: bump curve25519-dalek from 3.1.0 to 3.2.0 ( #19051 )
...
* chore: bump curve25519-dalek from 3.1.0 to 3.2.0
Bumps [curve25519-dalek](https://github.com/dalek-cryptography/curve25519-dalek ) from 3.1.0 to 3.2.0.
- [Release notes](https://github.com/dalek-cryptography/curve25519-dalek/releases )
- [Changelog](https://github.com/dalek-cryptography/curve25519-dalek/blob/main/CHANGELOG.md )
- [Commits](https://github.com/dalek-cryptography/curve25519-dalek/compare/3.1.0...3.2.0 )
---
updated-dependencies:
- dependency-name: curve25519-dalek
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-08-04 09:14:32 -06:00
999fb4bda9
chore: bump sysctl from 0.4.1 to 0.4.2 ( #19052 )
...
Bumps [sysctl](https://github.com/johalun/sysctl-rs ) from 0.4.1 to 0.4.2.
- [Release notes](https://github.com/johalun/sysctl-rs/releases )
- [Changelog](https://github.com/johalun/sysctl-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/johalun/sysctl-rs/commits/v0.4.2 )
---
updated-dependencies:
- dependency-name: sysctl
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-04 09:09:08 -06:00
2b33c0c165
stake: Allow stakes with unmatched credits observed to merge ( #18985 )
...
* stake: Allow stakes with unmatched credits observed to merge
* Address feedback
* Remove branch by doing a ceiling in one calc
2021-08-04 10:43:34 -04:00
31a620c42b
move towards accounts index being dynamic ( #19032 )
2021-08-04 09:18:05 -05:00
06e08c4840
move package_snapshots to AccountsPackagePre ctors ( #18997 )
...
This PR solves #18815 . Note that I had to make the snapshot prefix
constants inside `snapshot_utils.rs` public at the crate level in order
to make this work. I'm not sure whether or not introducing this
dependency is entirely good, either way the `snapshot_utils.rs` file
needs a lot of rework so things will move around, I believe this does
the work in the meantime. Any feedback will be greatly appreciated.
2021-08-04 09:03:03 -05:00
f7be47b8b9
chore:(deps): bump @project-serum/serum in /explorer ( #19048 )
...
Bumps [@project-serum/serum](https://github.com/project-serum/serum-ts ) from 0.13.54 to 0.13.55.
- [Release notes](https://github.com/project-serum/serum-ts/releases )
- [Commits](https://github.com/project-serum/serum-ts/commits )
---
updated-dependencies:
- dependency-name: "@project-serum/serum"
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-04 08:08:03 +00:00
0a63f65c03
Bumps solana_rbpf to v0.2.14 ( #18869 )
...
* Bumps solana_rbpf to v0.2.14
* Feature gate for verify_mul64_imm_nonzero as discussed in #17520 .
2021-08-04 09:50:28 +02:00
9912b73f9f
ci: expand acceptable grcov version regex
2021-08-04 07:21:55 +00:00
38fb6dcf86
bump rust stable to 1.54, nightly 2021-08-02
2021-08-04 07:21:55 +00:00
06a7a9e544
remove superfluous `collect()`s
2021-08-04 07:21:55 +00:00
79b4b83d3c
runtime: remove unused `allow` attrs
2021-08-04 07:21:55 +00:00
0b7ed18cfa
recvmmsg IPv6 awareness ( #18957 )
2021-08-03 20:35:50 -07:00
d8984cb0f4
chore: bump rollup from 2.55.0 to 2.55.1 in /web3.js ( #19038 )
...
Bumps [rollup](https://github.com/rollup/rollup ) from 2.55.0 to 2.55.1.
- [Release notes](https://github.com/rollup/rollup/releases )
- [Changelog](https://github.com/rollup/rollup/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rollup/rollup/compare/v2.55.0...v2.55.1 )
---
updated-dependencies:
- dependency-name: rollup
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-03 21:24:06 +00:00
13110c16de
chore: bump typedoc from 0.21.4 to 0.21.5 in /web3.js ( #19037 )
...
Bumps [typedoc](https://github.com/TypeStrong/TypeDoc ) from 0.21.4 to 0.21.5.
- [Release notes](https://github.com/TypeStrong/TypeDoc/releases )
- [Commits](https://github.com/TypeStrong/TypeDoc/compare/v0.21.4...v0.21.5 )
---
updated-dependencies:
- dependency-name: typedoc
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-03 21:01:59 +00:00
b8bb5b0c06
chore: bump @babel/preset-env from 7.14.8 to 7.14.9 in /web3.js ( #19035 )
...
Bumps [@babel/preset-env](https://github.com/babel/babel/tree/HEAD/packages/babel-preset-env ) from 7.14.8 to 7.14.9.
- [Release notes](https://github.com/babel/babel/releases )
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md )
- [Commits](https://github.com/babel/babel/commits/v7.14.9/packages/babel-preset-env )
---
updated-dependencies:
- dependency-name: "@babel/preset-env"
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-03 20:49:03 +00:00
Jeff Washington (jwash)
behzad nouri
Brian Anderson
dependabot[bot]
sakridge
Brooks Prumo
Trent Nelson
Jon Cinque
Oliver
Alexander Meißner
Jeff Biseda