Commit Graph

21703 Commits

Author SHA1 Message Date
Steven Luscher 3c01f4dd76
chore: remove Explorer build scripts from the Monorepo (#30262)
* chore: remove Explorer build scripts from the Monorepo

* chore: remove labeler action
2023-02-10 17:25:32 -06:00
Jeff Washington (jwash) 51d354defb
simplify AccountsToCombine.accounts_keep_slots (#30246) 2023-02-10 13:53:03 -08:00
Jeff Washington (jwash) a46ba32e9a
remove unused metric (#30248) 2023-02-10 13:36:22 -08:00
dependabot[bot] 66dd9432f5
Bump futures-util from 0.3.25 to 0.3.26 (#30240)
* Bump futures-util from 0.3.25 to 0.3.26

Bumps [futures-util](https://github.com/rust-lang/futures-rs) from 0.3.25 to 0.3.26.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.25...0.3.26)

---
updated-dependencies:
- dependency-name: futures-util
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2023-02-10 13:46:16 -07:00
Brooks b3c2256799
Refactors a sort's comparator fn (#30243) 2023-02-10 15:41:55 -05:00
steviez 5ad8c8bd20
fixup! lt: Reallow custom accounts path with Secondary access (#30228) (#30236)
The previous change used Primary access; however, it is better to use
PrimaryForMaintenance in this case to disallow any potential compaction
from happening.
2023-02-10 14:09:14 -06:00
Jeff Washington (jwash) 33cc4cc3d5
maintain sorted order on ancient target slots (#30247) 2023-02-10 12:08:13 -08:00
behzad nouri ded457cd73
embeds the new gossip ContactInfo in ClusterInfo (#30022)
Working towards replacing the legacy gossip contact-info with the new
one, the commit updates the respective field in gossip cluster-info.
2023-02-10 20:07:45 +00:00
Jeff Washington (jwash) 157a9d725f
add ancient pack_ancient_storages (#30235)
* add ancient pack_ancient_storages

* cleanup
2023-02-10 12:06:36 -08:00
behzad nouri ceb225f36e
caches blockstore duplicate shreds slots (#30241)
In order to reduce number of calls to blockstore, the commit locally caches
blockstore.has_duplicate_shreds_in_slot in DuplicateShredHandler.consumed.
2023-02-10 18:19:57 +00:00
HaoranYi 0b5ae8abe1
Refactor - simplify geyser notify book keeping code and fix total_accounts stat (#30207)
* refactor common code

* fix total_accounts stat

* clippy
2023-02-10 12:18:50 -06:00
Jeff Washington (jwash) 12411e5215
pass shrink_can_be_active through shrink fns (#30244) 2023-02-10 10:06:40 -08:00
Lijun Wang 84fbf9273b
Refactor thin client (#30229)
Make client/thin_client a thin wrapper and forward calls to the thin-client/thin_client
2023-02-10 09:55:13 -08:00
Jeff Washington (jwash) dcb2d6c8ae
collect_sort_filter_ancient_slots takes tuning by ref (#30242) 2023-02-10 09:40:13 -08:00
Jeff Washington (jwash) 0113fe340e
add ancient write_ancient_accounts_multiple_refs (#30198) 2023-02-10 09:39:30 -08:00
Denis K c078ffd9ee
Added support of consumed BPF units tracing (#30042)
* Added support of consumed BPF units tracing

* Fixed "Inconsistent trace log stack" when logging units before first instruction trace

* Rewrote the solution in order to reuse `trace_log_stack` and check whether tracing is enabled

* Instruction tracing is disabled by default
2023-02-10 16:17:44 +01:00
Brooks d27c860249
Replaces stable sort with unstable sort (#30223) 2023-02-10 09:56:11 -05:00
Jeff Washington (jwash) 128ffbdf9b
restore test to all slot combos (#30231) 2023-02-10 06:25:37 -08:00
Ryo Onodera 3e6162e69e
Add address lookup tables to minimized snapshot (#30158)
* Add address lookup tables to minimized snapshot

* Add comment for future posterity

* Add reference to the issue

* Adjust comment a bit

Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>

---------

Co-authored-by: Andrew Fitzgerald <apfitzge@gmail.com>
2023-02-10 14:46:02 +09:00
Andrew Fitzgerald 60cf8ce65b
remove unnecessary lifetime (#30108)
Remove unnecessary lifetime on function
2023-02-09 21:17:41 -08:00
steviez 64d2809244
ledger-tool: Reallow custom accounts path with Secondary access (#30228)
Previously, ledger-tool had a guardrail to disallow a custom accounts
path when access mode to the blockstore was Secondary. This was to avoid
potentially pulling the accounts out from underneath solana-validator.
When ledger-tool switched over to use Secondary blockstore access for
all commands that do not need write access, this removed the ability to
use custom accounts paths with ledger-tool at all for these commands.
Custom accounts paths are desirable, especially if that custom path is
in tmpfs to speed up processing.

With this change, when a custom accounts path is passed for a command
using Secondary access, ledger-tool now checks if Primary access is
being held by another process. If not, allow processing to proceed
with the custom accounts path.

The above check isn't fullproof, but it is about equal to the check that
previously existed when ledger-tool would run in Primary access mode
when it didn't need to.
2023-02-09 19:53:31 -06:00
Tyera 45272147ad
Fill in feature issue number in feature_set (#30232)
Fill in feature issue number
2023-02-10 01:14:25 +00:00
Jeff Washington (jwash) fc8c590375
improve debugging test fn compare_all_accounts (#30218) 2023-02-09 15:06:09 -08:00
Jeff Washington (jwash) 639f3475f5
test and comment cleanup (#30226)
* test and comment cleanup

* pr feedback
2023-02-09 15:02:10 -08:00
Jeff Biseda 180273b97d
defer HighestShred repairs during shred propagation threshold (#30142) 2023-02-09 14:57:55 -08:00
Jeff Washington (jwash) db25ccba52
ancient add write_one_packed_storage (#30220) 2023-02-09 14:47:22 -08:00
Ashwin Sekar 67f644473b
Fix repair behavior concerning our own leader slots (#30200)
panic when trying to dump & repair a block that we produced
2023-02-09 14:30:12 -07:00
Andrew Fitzgerald 4b17acf64e
BankingStage Refactor: Add state to Committer (#30107) 2023-02-09 13:22:42 -08:00
Benno Fünfstück e8c43aa0d1
ledger-tool: Notify geyser of transactions (#29933)
ledger-tool: Notify geyser of transactions
2023-02-09 12:34:20 -08:00
ivandzen 2f9146e8c8
Replace txn_signature in ReplicaAccountInfo with transaction (#30189)
#### Current implementation of ReplicaAccountInfo restricts
possibilities for inflight account filtration
Current implementation includes transaction signature in
ReplicaAccountInfo. This approach does not allow to filter accounts by
matching other accounts in transaction in-flight (e. g. accept only
those accounts included in transactions for specific programs). Current
implementation forces to collect ALL accounts and transactions for some
period of time and perform such complex filtration afterwards.

#### Pass reference to transaction object instead of transaction
signature into ReplicaAccountInfo
Advanced in-flight filtration can be implemented in plugins by passing
reference to transaction for every update_account event. This change
doesn't bring any overhead comparing to current implementation (only
data type of reference is changed) and brings only minor changes in
source code.
2023-02-09 12:00:33 -08:00
Brennan da34f2edd0
Fix duplicate broadcast test (#30209) 2023-02-09 11:58:54 -08:00
Jeff Washington (jwash) c6695996b8
add test parameters to fix build (#30227)
#### Problem
build broken due to pr conflicts

#### Summary of Changes


Fixes #
<!-- OPTIONAL: Feature Gate Issue: # -->
<!-- Don't forget to add the "feature-gate" label -->
2023-02-09 11:56:12 -08:00
Jeff Washington (jwash) 62b93df1c7
change ancient tests to correct size (#30221)
#### Problem
fix tests to be more realistic

#### Summary of Changes
allocate correctly sized append vecs

Fixes #
<!-- OPTIONAL: Feature Gate Issue: # -->
<!-- Don't forget to add the "feature-gate" label -->
2023-02-09 11:41:36 -08:00
Jeff Washington (jwash) 99937af69d
test fn append_single_account_with_default_hash can add to index (#30219)
#### Problem
When adding test accounts to storages, it is useful to be able to make
the accounts alive in the index, too.

#### Summary of Changes
test fn append_single_account_with_default_hash can add to index

Fixes #
<!-- OPTIONAL: Feature Gate Issue: # -->
<!-- Don't forget to add the "feature-gate" label -->
2023-02-09 11:37:02 -08:00
Jeff Washington (jwash) e0d354f542
fix bug in test fn compare_all_accounts (#30217)
#### Problem
test code compares sets of accounts after clean/shrink operations.
The indexing was off and new tests were giving false positives.
The test code was equivalent as long as everything was in the same
order.

#### Summary of Changes
Fix test code comparison indexes.

Fixes #
<!-- OPTIONAL: Feature Gate Issue: # -->
<!-- Don't forget to add the "feature-gate" label -->
2023-02-09 11:28:53 -08:00
dependabot[bot] 57dced05f0
Bump sha2 from 0.10.5 to 0.10.6 (#30215)
Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.10.5 to
0.10.6.

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2023-02-09 12:18:09 -07:00
Jeff Washington (jwash) 68b2b1badd
add ancient calc_accounts_to_combine (#30195)
#### Problem
Building new algorithm for packing ancient storage. Packing will occur
in 1 pass across multiple ancient slots.
This will be put in 1 dead code piece at a time with tests until all
pieces are present. Switch between current packing algorithm and this
new one is in a validator cli argument. Resulting append vecs are
correct and compatible (as a set) either way. When a new storage format
optimized for cold storage becomes available, it will only work with
this new packing algorithm, so the change will need to be complete prior
to the new storage format.

#### Summary of Changes
Add `ancient calc_accounts_to_combine` to separate accounts to prepare
for creating packed ancient append vecs. This will be used soon.

Fixes #
<!-- OPTIONAL: Feature Gate Issue: # -->
<!-- Don't forget to add the "feature-gate" label -->
2023-02-09 11:06:17 -08:00
dependabot[bot] 2b4a6a4dd9
Bump libloading from 0.7.3 to 0.7.4 (#30216)
Bumps [libloading](https://github.com/nagisa/rust_libloading) from 0.7.3
to 0.7.4.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="6e284984ae"><code>6e28498</code></a>
Fix CI</li>
<li><a
href="95a0f62e8f"><code>95a0f62</code></a>
Placate clippy</li>
<li><a
href="224a3def35"><code>224a3de</code></a>
Release 0.7.4</li>
<li><a
href="bd17713dcc"><code>bd17713</code></a>
Support AIX dyld constants</li>
<li><a
href="6e07929736"><code>6e07929</code></a>
fix typo</li>
<li><a
href="6b33651a90"><code>6b33651</code></a>
Construct a PathBuf</li>
<li><a
href="964dd8b4c0"><code>964dd8b</code></a>
Simplify lib_path</li>
<li><a
href="8009f51a9f"><code>8009f51</code></a>
Honor Cargo's target directory when building LIBPATH</li>
<li><a
href="842e377ae4"><code>842e377</code></a>
Fix a typo in the CI scripts</li>
<li>See full diff in <a
href="https://github.com/nagisa/rust_libloading/compare/0.7.3...0.7.4">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=libloading&package-manager=cargo&previous-version=0.7.3&new-version=0.7.4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2023-02-09 10:53:18 -07:00
Andrew Fitzgerald 058738424d
BankingStage Refactor: transaction recorder record transactions (#30106) 2023-02-09 08:34:02 -08:00
behzad nouri b3887af7c6
flattens the internal buffer in the duplicate-shreds handler (#30196)
Duplicate-shreds handler is using a nested hash-map for the incomplete
chunks buffered. This is resulting in a convoluted logic to limit the
number of entries:
https://github.com/solana-labs/solana/blob/427bd6264/gossip/src/duplicate_shred_handler.rs#L62

This commit instead uses a flat buffer mapping (Slot, Pubkey) pairs to
the respective duplicate shreds chunks. The buffer is allowed to grow to
twice the intended capacity, at which point the extraneous entries are
removed in linear time, resulting an amortized O(1) performance.
2023-02-09 16:33:20 +00:00
Jeff Washington (jwash) a4fb1834d1
AliveAccounts includes source slot (#30208) 2023-02-09 05:57:58 -08:00
Alexander Meißner f7dfe8337a
Moves `if use_jit` outside of `#[cfg()]`. (#30202) 2023-02-09 09:14:26 +01:00
dependabot[bot] 4d599a8362
Bump rand_core from 0.6.3 to 0.6.4 (#30114)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2023-02-08 21:27:26 -07:00
dependabot[bot] 46293ec43f
Bump openssl from 0.10.41 to 0.10.45 (#30115)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2023-02-08 21:26:43 -07:00
Nick Frostbutter 4a10ebdf0b
[docs] style: prettier force line length to 80 chars (#30133) 2023-02-09 03:23:21 +00:00
Nick Frostbutter ae59195ed0
[docs] fix: rpc api redirect (#30212) 2023-02-08 20:40:05 -06:00
steviez d3dab24bbe
chore: Use `i` over `ix` variable name when naming worker threads (#30206) 2023-02-09 01:24:57 +00:00
behzad nouri 1ad69cfc38
removes dynamic cast and dynamic dispatch from connection-cache (#30128)
Dynamic dispatch forces heap allocation and adds extra overhead.
Dynamic casting as in the ones below, lacks compile-time type safety:
https://github.com/solana-labs/solana/blob/eeb622c4e/quic-client/src/lib.rs#L172-L175
https://github.com/solana-labs/solana/blob/eeb622c4e/udp-client/src/lib.rs#L52-L55

The commit removes all instances of Any, Box<dyn ...>, and Arc<dyn ...>,
and instead uses generic and associated types.

There are only two protocols QUIC and UDP; and the code which has to
work with both protocols can use a trivial thin enum wrapper.

With respect to connection-cache specifically:
* connection-cache/ConnectionCache is a single protocol cache which
  allows to use either QUIC or UDP without any build dependency on the
  other protocol.
* client/ConnectionCache is an enum wrapper around both protocols and
  can be used in the code which has to work with both QUIC and UDP.

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2023-02-09 00:50:44 +00:00
Jon Cinque 0481ce3069
sdk: Make `declare_id` mostly const (#30177) 2023-02-09 01:26:09 +01:00
steviez 55c8963462
Append integer to solBigTGetBlk threads to uniquify them (#30184) 2023-02-09 00:23:29 +00:00