Go to file
apfitzge f4189c0305
ledger-tool minimized snapshots (#25334)
* working on local snapshot

* Parallelization for slot storage minimization

* Additional clean-up and fixes

* make --minimize an option of create-snapshot

* remove now unnecessary function

* Parallelize parts of minimized account set generation

* clippy fixes

* Add rent collection accounts and voting node_pubkeys

* Simplify programdata_accounts generation

* Loop over storages to get slot set

* Parallelize minimized slot set generation

* Parallelize adding owners and programdata_accounts

* Remove some now unncessary checks on the blockstore

* Add a warning for minimized snapshots across epoch boundary

* Simplify ledger-tool minimize

* Clarify names of bank's minimization helper functions

* Remove unnecesary funciton, fix line spacing

* Use DashSets instead of HashSets for minimized account and slot sets

* Filter storages uses all threads instead of thread_pool

* Add some additional comments on functions for minimization

* Moved more into bank and parallelized

* Update programs/bpf/Cargo.lock for dashmap in ledger

* Clippy fix

* ledger-tool: convert minimize_bank_for_snapshot Measure into measure!

* bank.rs: convert minimize_bank_for_snapshot Measure into measure!

* accounts_db.rs: convert minimize_accounts_db Measure into measure!

* accounts_db.rs: add comment about use of minimize_accounts_db

* ledger-tool: CLI argument clarification

* minimization functions: make infos unique

* bank.rs: Add test_get_rent_collection_accounts_between_slots

* bank.rs: Add test_minimization_add_vote_accounts

* bank.rs: Add test_minimization_add_stake_accounts

* bank.rs: Add test_minimization_add_owner_accounts

* bank.rs: Add test_minimization_add_programdata_accounts

* accounts_db.rs: Add test_minimize_accounts_db

* bank.rs: Add negative case and comments in test_get_rent_collection_accounts_between_slots

* bank.rs: Negative test in test_minimization_add_programdata_accounts

* use new static runtime and sdk ids

* bank comments to doc comments

* Only need to insert the maximum slot a key is found in

* rename remove_pubkeys to purge_pubkeys

* add comment on builtins::get_pubkeys

* prevent excessive logging of removed dead slots

* don't need to remove slot from shrink slot candidates

* blockstore.rs: get_accounts_used_in_range shouldn't return Result

* blockstore.rs: get_accounts_used_in_range: parallelize slot loop

* report filtering progress on time instead of count

* parallelize loop over snapshot storages

* WIP: move some bank minimization functionality into a new class

* WIP: move some accounts_db minimization functionality into SnapshotMinimizer

* WIP: Use new SnapshotMinimizer

* SnapshotMinimizer: fix use statements

* remove bank and accounts_db minimization code, where possible

* measure! doesn't take a closure

* fix use statement in blockstore

* log_dead_slots does not need pub(crate)

* get_unique_accounts_from_storages does not need pub(crate)

* different way to get stake accounts/nodes

* fix tests

* move rent collection account functionality to snapshot minimizer

* move accounts_db minimize behavior to snapshot minimizer

* clean up

* Use bank reference instead of Arc. Additional comments

* Add a comment to blockstore function

* Additional clarifying comments

* Moved all non-transaction account accumulation into the SnapshotMinimizer.

* transaction_account_set does not need to be mutable now

* Add comment about load_to_collect_rent_eagerly

* Update log_dead_slots comment

* remove duplicate measure/print of get_minimized_slot_set
2022-06-22 13:17:43 -04:00
.buildkite
.github
account-decoder Remove some clippy lints 2022-06-22 09:23:22 -07:00
accounts-bench Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
accounts-cluster-bench Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
banking-bench Use single send socket in UdpTpuConnection (#26105) 2022-06-21 14:56:21 -07:00
banks-client Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
banks-interface Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
banks-server Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
bench-streamer Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
bench-tps Use single send socket in UdpTpuConnection (#26105) 2022-06-21 14:56:21 -07:00
bloom Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
bucket_map Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
ci Remove some clippy lints 2022-06-22 09:23:22 -07:00
clap-utils Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
clap-v3-utils Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
cli Remove some clippy lints 2022-06-22 09:23:22 -07:00
cli-config Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
cli-output Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
client Remove some clippy lints 2022-06-22 09:23:22 -07:00
client-test Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
core Remove some clippy lints 2022-06-22 09:23:22 -07:00
docs Add `prioritization fee` to terminology (#26093) 2022-06-22 10:19:01 -05:00
dos Use single send socket in UdpTpuConnection (#26105) 2022-06-21 14:56:21 -07:00
download-utils Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
entry Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
explorer
faucet Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
frozen-abi Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
genesis Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
genesis-utils Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
geyser-plugin-interface Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
geyser-plugin-manager Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
gossip Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
install Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
keygen Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
ledger ledger-tool minimized snapshots (#25334) 2022-06-22 13:17:43 -04:00
ledger-tool ledger-tool minimized snapshots (#25334) 2022-06-22 13:17:43 -04:00
local-cluster Use single send socket in UdpTpuConnection (#26105) 2022-06-21 14:56:21 -07:00
log-analyzer Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
logger Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
measure Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
merkle-root-bench Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
merkle-tree Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
metrics Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
multinode-demo
net Add profile option to net scripts (#25624) 2022-06-21 19:28:15 +02:00
net-shaper Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
net-utils Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
notifier Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
perf Remove some clippy lints 2022-06-22 09:23:22 -07:00
poh Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
poh-bench Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
program-runtime Remove some clippy lints 2022-06-22 09:23:22 -07:00
program-test Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
programs ledger-tool minimized snapshots (#25334) 2022-06-22 13:17:43 -04:00
rayon-threadlimit Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
rbpf-cli Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
remote-wallet Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
rpc Remove some clippy lints 2022-06-22 09:23:22 -07:00
rpc-test Use single send socket in UdpTpuConnection (#26105) 2022-06-21 14:56:21 -07:00
runtime ledger-tool minimized snapshots (#25334) 2022-06-22 13:17:43 -04:00
scripts
sdk Remove some clippy lints 2022-06-22 09:23:22 -07:00
send-transaction-service Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
stake-accounts Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
storage-bigtable Add metric for tracking Bigtable block uploads (#26027) 2022-06-21 18:27:02 -06:00
storage-proto Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
streamer QUIC stream timeouts if no data is received (#26116) 2022-06-21 18:56:47 -07:00
sys-tuner Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
system-test
test-validator Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
tokens Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
transaction-dos Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
transaction-status Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
upload-perf Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
validator Remove some clippy lints 2022-06-22 09:23:22 -07:00
version Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
watchtower Bump version to 1.11.1 (#26104) 2022-06-21 12:07:46 -05:00
web3.js chore: fix types of `RpcRequest` and `RpcBatchRequest` (#26102) 2022-06-21 14:48:44 -07:00
zk-token-sdk Remove some clippy lints 2022-06-22 09:23:22 -07:00
.clippy.toml
.codecov.yml
.gitignore
.mergify.yml
.travis.yml
CONTRIBUTING.md
Cargo.lock ledger-tool minimized snapshots (#25334) 2022-06-22 13:17:43 -04:00
Cargo.toml
LICENSE
README.md
RELEASE.md
SECURITY.md
cargo
cargo-build-bpf
cargo-build-sbf
cargo-test-bpf
cargo-test-sbf
fetch-perf-libs.sh
fetch-spl.sh
run.sh
rustfmt.toml
test-abi.sh
vercel.json

README.md

Solana

Solana crate Solana documentation Build status codecov

Building

1. Install rustc, cargo and rustfmt.

$ curl https://sh.rustup.rs -sSf | sh
$ source $HOME/.cargo/env
$ rustup component add rustfmt

When building the master branch, please make sure you are using the latest stable rust version by running:

$ rustup update

When building a specific release branch, you should check the rust version in ci/rust-version.sh and if necessary, install that version by running:

$ rustup install VERSION

Note that if this is not the latest rust version on your machine, cargo commands may require an override in order to use the correct version.

On Linux systems you may need to install libssl-dev, pkg-config, zlib1g-dev, protobuf etc. On Ubuntu:

$ sudo apt-get update
$ sudo apt-get install libssl-dev libudev-dev pkg-config zlib1g-dev llvm clang cmake make libprotobuf-dev protobuf-compiler

2. Download the source code.

$ git clone https://github.com/solana-labs/solana.git
$ cd solana

3. Build.

$ cargo build

Testing

Run the test suite:

$ cargo test

Starting a local testnet

Start your own testnet locally, instructions are in the online docs.

Accessing the remote development cluster

  • devnet - stable public cluster for development accessible via devnet.solana.com. Runs 24/7. Learn more about the public clusters

Benchmarking

First, install the nightly build of rustc. cargo bench requires the use of the unstable features only available in the nightly build.

$ rustup install nightly

Run the benchmarks:

$ cargo +nightly bench

Release Process

The release process for this project is described here.

Code coverage

To generate code coverage statistics:

$ scripts/coverage.sh
$ open target/cov/lcov-local/index.html

Why coverage? While most see coverage as a code quality metric, we see it primarily as a developer productivity metric. When a developer makes a change to the codebase, presumably it's a solution to some problem. Our unit-test suite is how we encode the set of problems the codebase solves. Running the test suite should indicate that your change didn't infringe on anyone else's solutions. Adding a test protects your solution from future changes. Say you don't understand why a line of code exists, try deleting it and running the unit-tests. The nearest test failure should tell you what problem was solved by that code. If no test fails, go ahead and submit a Pull Request that asks, "what problem is solved by this code?" On the other hand, if a test does fail and you can think of a better way to solve the same problem, a Pull Request with your solution would most certainly be welcome! Likewise, if rewriting a test can better communicate what code it's protecting, please send us that patch!

Disclaimer

All claims, content, designs, algorithms, estimates, roadmaps, specifications, and performance measurements described in this project are done with the Solana Foundation's ("SF") good faith efforts. It is up to the reader to check and validate their accuracy and truthfulness. Furthermore, nothing in this project constitutes a solicitation for investment.

Any content produced by SF or developer resources that SF provides are for educational and inspirational purposes only. SF does not encourage, induce or sanction the deployment, integration or use of any such applications (including the code comprising the Solana blockchain protocol) in violation of applicable laws or regulations and hereby prohibits any such deployment, integration or use. This includes the use of any such applications by the reader (a) in violation of export control or sanctions laws of the United States or any other applicable jurisdiction, (b) if the reader is located in or ordinarily resident in a country or territory subject to comprehensive sanctions administered by the U.S. Office of Foreign Assets Control (OFAC), or (c) if the reader is or is working on behalf of a Specially Designated National (SDN) or a person subject to similar blocking or denied party prohibitions.

The reader should be aware that U.S. export control and sanctions laws prohibit U.S. persons (and other persons that are subject to such laws) from transacting with persons in certain countries and territories or that are on the SDN list. As a project-based primarily on open-source software, it is possible that such sanctioned persons may nevertheless bypass prohibitions, obtain the code comprising the Solana blockchain protocol (or other project code or applications) and deploy, integrate, or otherwise use it. Accordingly, there is a risk to individuals that other persons using the Solana blockchain protocol may be sanctioned persons and that transactions with such persons would be a violation of U.S. export controls and sanctions law. This risk applies to individuals, organizations, and other ecosystem participants that deploy, integrate, or use the Solana blockchain protocol code directly (e.g., as a node operator), and individuals that transact on the Solana blockchain through light clients, third party interfaces, and/or wallet software.