Commit Graph

24219 Commits

Author SHA1 Message Date
Brooks 8fbe033eac
Moves val into .extend() instead of &mut and .into_iter() (#34285) 2023-12-01 14:38:01 -05:00
steviez c9cb727b1f
Add additional context to ReplayStage log message (#34296)
The pre-existing message was somewhat vague, so give more details about the caller
2023-12-01 13:04:39 -06:00
Brooks 4f65f7dc8d
Sanitizes tiered storage footer after reading from disk (#34200) 2023-12-01 13:58:10 -05:00
Yueh-Hsuan Chiang fa0930f254
[TieredStorage] Allow HotStorage to handle more account data (#34155)
#### Problem
After we have defined AccountOffset to be u32, it means the address space within
one HotAccountsFile is up to 4GB.  However, since the Accounts Blocks in a
HotAccountsFile is 8-byte aligned, it has the opportunity to store more data by
internally multiplying the AccountOffset by 8.

#### Summary of Changes
This PR allows a HotAccountsFile to store up to 32GB accounts data by using
8 x AccountOffset as its actual offset.

#### Test Plan
Updated existing unit-tests.
2023-12-01 10:39:15 -08:00
Lucas Steuernagel b97b3dd4ab
Use BankForks on tests - Part 3 (#34248)
* Add BankForks to core tests

* Refactor functions under DCOU
2023-12-01 13:47:22 -03:00
Andrew Fitzgerald 2294801954
Do not derive Copy for EpochSchedule and Rent (#32767) 2023-12-01 07:57:25 -08:00
Tyera 68928bb7d6
solana-cli: format strings (#34288)
* Format strings in solana-cli

* Restore some wallet.rs strings that were better before

* Fixup vote.rs string

* Fixup stake.rs string

* Fixup program.rs strings

* Fixup some address_lookup_table.rs strings

* Fixup clap_app.rs string

* Clarify validator_info help

* Fixup cluster_query.rs
2023-12-01 09:00:14 +00:00
steviez 7c4e7230b0
ledger-tool: Run rustfmt with format_strings = true (#34284)
Long string literals can cause rustfmt to fail, which results in rustfmt
failing to format entire functions. There are several instances of this
in ledger-tool, so format these files with wrapped strings so that
formatting will apply to functions again.

Note that this PR was created by adding format_strings = true to
rustfmt.toml; however, this change does NOT persist that rule as the
rule would format the entire repo.
2023-12-01 02:50:18 -06:00
Ashwin Sekar d84dcd37bc
blockstore: use u32 for fec_set_index in erasure set index store key (#34268)
* blockstore: use u32 for fec_set_index in erasure set index store key

* pr feedback u64::from
2023-11-30 19:17:49 -05:00
Andrew Fitzgerald 18309ba8da
TransactionScheduler: Schedule Filter (#34252) 2023-11-30 14:41:11 -08:00
steviez 479b7ee9f2
Bubble up errors in bank_fork_utils instead of exiting process (#34277)
There are operations in bank_fork_utils that may fail; we explicitly
call std::process::exit() on several of these. Granted we may end up
exiting the process higher up the callstack, bubbling the errors up
allow a caller that could handle the error to do so.
2023-11-30 16:35:59 -06:00
Tyera c5368a3d35
ledger-tool: read bigtable entries (#34266)
* Add helper to get a protobuf cell without bothering with bincode

* Add conversion back to EntrySummaries

* Add LedgerStorage::get_block_entries

* Add ledger-tool bigtable entries subcommand

* Move CliEntries into ledger_tool::output for now

* Fetch entries with read-only token
2023-11-30 21:33:04 +00:00
steviez 71c1782c74
Allow Blockstore to open unknown columns (#34174)
As we develop new features or modifications, we occassionally need to
introduce new columns to the Blockstore. Adding a new column introduces
a compatibility break given that opening the database in Primary mode
(R/W access) requires opening all columns. Reverting to an old software
version that is unaware of the new column is obviously problematic.

In the past, we have addressed by backporting minimal "stub" PR's to
older versions. This is annoying, and only allow compatibility for the
single version or two that we backport to.

This PR adds a change to automatically detect all columns, and create
default column descriptors for columns we were unaware of. As a result,
older software versions can open a Blockstore that was modified by a
newer software version, even if that new version added columns that the
old version is unaware of.
2023-11-30 13:24:56 -06:00
Tyera 834a3968ef
Fix patch comment typos and wrap lines (#34251)
* Comment nits

* Wrap to 80 chars

* Wrap some more comment lines
2023-11-30 12:18:54 -07:00
Yihau Chen 4832b4ecec
ci: add macos clippy test (#34272)
* ci: add clippy test for macos and windows

* ci: remove windows from clippy test

* ci: cancel cargo clippy on Github Actions when new commits comming

* ci: set -e for .github/scripts/cargo-clippy-before-script.sh
2023-12-01 00:59:45 +08:00
Tyera 5085ee9b1a
Only compile bench_program_alu for x86_64 (#34282)
Ignore bench if not x86_64
2023-11-30 07:48:08 -07:00
Tao Zhu c3323c0c36
Add test module to runtime-transaction (#34273)
* basic tests
2023-11-29 18:30:59 -06:00
steviez 935e06f8f1
Output BankHashDetails file when leader drops its' own block (#34256)
Currently, the file is generated when a node drops a block that was
produced by another node. However, it would also be beneficial to see
the account state when a node drops its' own block.

Output the file in this additional failure codepath
2023-11-29 17:20:27 -06:00
Andrew Fitzgerald e949ef9daa
Trailing _us for scheduler time metrics (#34263) 2023-11-29 13:18:56 -08:00
Lijun Wang f211c86e69
Support logging in geyser plugins (#34101)
The change creates a new interface in the GeyserPlugin interface to allow the runtime to pass the logging configuration to the plugin.
2023-11-29 11:31:11 -08:00
Ashwin Sekar 60c09d3d89
local-cluster: ignore test_duplicate_shreds_switch_failure (#34267) 2023-11-29 17:34:57 +00:00
Lucas Steuernagel aeb4a348fb
Use BankForks on tests - Part 2 (#34234)
* Use BankForks on tests - Part 2

Signed-off-by: Lucas Steuernagel <lucas.tnagel@gmail.com>
2023-11-29 14:28:31 -03:00
Ashwin Sekar e1165aaf00
blockstore: populate merkle root metas column (#34097) 2023-11-29 11:14:24 -05:00
dependabot[bot] 1f00f8e29c
build(deps): bump js-sys from 0.3.65 to 0.3.66 (#34244)
* build(deps): bump js-sys from 0.3.65 to 0.3.66

Bumps [js-sys](https://github.com/rustwasm/wasm-bindgen) from 0.3.65 to 0.3.66.
- [Release notes](https://github.com/rustwasm/wasm-bindgen/releases)
- [Changelog](https://github.com/rustwasm/wasm-bindgen/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rustwasm/wasm-bindgen/commits)

---
updated-dependencies:
- dependency-name: js-sys
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2023-11-29 13:27:11 +00:00
dependabot[bot] 0c4abcbcc2
build(deps): bump wasm-bindgen from 0.2.88 to 0.2.89 (#34243)
* build(deps): bump wasm-bindgen from 0.2.88 to 0.2.89

Bumps [wasm-bindgen](https://github.com/rustwasm/wasm-bindgen) from 0.2.88 to 0.2.89.
- [Release notes](https://github.com/rustwasm/wasm-bindgen/releases)
- [Changelog](https://github.com/rustwasm/wasm-bindgen/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rustwasm/wasm-bindgen/compare/0.2.88...0.2.89)

---
updated-dependencies:
- dependency-name: wasm-bindgen
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2023-11-29 13:26:57 +00:00
Tyera e425c1acaa
Fix mac build (#34264)
Allow dead code instead
2023-11-29 04:07:13 +00:00
Tyera ded307adb3
storage-bigtable: delete entry rows if they exist when deleting confirmed block (#34261)
Delete entry rows if they exist
2023-11-28 18:18:46 -07:00
Tyera 2bde5c3cb2
Tag InterestingLimit enum with target_os (#34259)
Only linux has interesting limits
2023-11-29 00:52:32 +00:00
Andrew Fitzgerald df8893772e
TransactionScheduler: Clean already processed or old transactions from container (#34233) 2023-11-28 16:25:12 -08:00
Greg Cusack 0a2ff8525a
Increase pull request clusterinfo probability (#34231)
* ensure new contactinfo propagated quicker when handling pull requests

* improve readability
2023-11-28 16:08:12 -08:00
Alessandro Decina a0df9def91
direct_mapping: disable stack frame gaps (#34254)
Gaps don't work with direct mapping which assumes a 1:1 mapping between virtual
and host addresses.
2023-11-29 09:54:48 +11:00
Tyera 573ec81fbb
storage-bigtable: Upload entries (#34099)
* Add entries table to bt init

* Add entries to storage-proto

* Use new Blockstore method in bigtable_upload

* Add LedgerStorage::upload_confirmed_block_with_entries and use in bigtable_upload

* Upload entries to bigtable
2023-11-28 11:47:22 -07:00
Brooks b9ef204faf
Fixes BankFrom.*Timings (#34249) 2023-11-28 13:05:54 -05:00
steviez 4d2018fc6e
Use Vec::with_capacity() to avoid reallocations with repeated push()'s (#34242)
The length of tx_cells is known before the loop that populates it, so we
can appropriately allocate the Vec immediatley to avoid any
reallocations.
2023-11-28 11:26:47 -06:00
Jeff Washington (jwash) 32c7acbf74
add slot to assert message (#34225) 2023-11-28 10:29:39 -06:00
Andrew Fitzgerald 40f5870a47
Refactor: check_transactions - allow borrowed transactions (#34232) 2023-11-28 08:24:16 -08:00
Andrew Fitzgerald 656ec4bdf0
Bump prio-graph to 0.2.0 (#34235) 2023-11-28 08:23:06 -08:00
Andrew Fitzgerald 449d375565
Add metric for number of unschedulable transactions (#34230) 2023-11-28 08:20:53 -08:00
dependabot[bot] b741c9ce4f
build(deps): bump openssl from 0.10.59 to 0.10.60 (#34214)
* build(deps): bump openssl from 0.10.59 to 0.10.60

Bumps [openssl](https://github.com/sfackler/rust-openssl) from 0.10.59 to 0.10.60.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-v0.10.59...openssl-v0.10.60)

---
updated-dependencies:
- dependency-name: openssl
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2023-11-28 10:41:44 +08:00
dependabot[bot] 419d048828
build(deps): bump raptorq from 1.7.0 to 1.8.0 (#34223)
Bumps [raptorq](https://github.com/cberner/raptorq) from 1.7.0 to 1.8.0.
- [Release notes](https://github.com/cberner/raptorq/releases)
- [Commits](https://github.com/cberner/raptorq/compare/v1.7.0...v1.8.0)

---
updated-dependencies:
- dependency-name: raptorq
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-28 10:41:35 +08:00
Kirill Fomichev 873bef9fef
bigtable: fix AccessToken issues (#34213)
* bigtable: fix AccessToken issue

* remove inner

* less changes

* fmt + drop lock
2023-11-27 16:46:49 -07:00
Andrew Fitzgerald 564f1a97e0
Update CHANGELOG for new scheduler support (#34181)
Update change log for new scheduler support
2023-11-27 22:07:18 +00:00
Alessandro Decina 09088822e7
direct_mapping: fix iter_memory_pair_chunks in reverse mode (#34204)
iter_memory_pair_chunks was iterating regions in reverse, but not memory
_within_ regions in reverse.

This commit fixes the issue and simplifies the implementation by removing
nested loops which made control flow hard to reason about.
2023-11-28 08:46:20 +11:00
Lijun Wang 8445246b8f
Added a comment about order of dropping plugin and library (#34160)
Added a comment about order of dropping plugin and library
2023-11-27 13:07:26 -08:00
dependabot[bot] a22487f444
build(deps): bump proc-macro2 from 1.0.69 to 1.0.70 (#34222)
* build(deps): bump proc-macro2 from 1.0.69 to 1.0.70

Bumps [proc-macro2](https://github.com/dtolnay/proc-macro2) from 1.0.69 to 1.0.70.
- [Release notes](https://github.com/dtolnay/proc-macro2/releases)
- [Commits](https://github.com/dtolnay/proc-macro2/compare/1.0.69...1.0.70)

---
updated-dependencies:
- dependency-name: proc-macro2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2023-11-27 13:59:42 -07:00
dependabot[bot] efa59dc33f
build(deps): bump url from 2.4.1 to 2.5.0 (#34216)
* build(deps): bump url from 2.4.1 to 2.5.0

Bumps [url](https://github.com/servo/rust-url) from 2.4.1 to 2.5.0.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/compare/v2.4.1...v2.5.0)

---
updated-dependencies:
- dependency-name: url
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2023-11-27 13:58:51 -07:00
Andrew Fitzgerald 005c825b5c
Validate account locks on buffering (#34229) 2023-11-27 11:21:10 -08:00
Brooks 5c7ab5dc08
ledger-tool does *not* fastboot by default (#34228) 2023-11-27 13:48:28 -05:00
Lucas Steuernagel e83276522a
Use `BankForks` on tests - Part 1 (#34206)
---------

Signed-off-by: Lucas Steuernagel <lucas.tnagel@gmail.com>
2023-11-27 14:10:46 -03:00
Brooks da9fad84b3
Skip shrink until startup verification is complete (#34209) 2023-11-27 10:57:37 -05:00