Go to file
Brooks Prumo fe9ee9134a
Make background services aware of incremental snapshots (#19401)
AccountsBackgroundService now knows about incremental snapshots.  It is
now also in charge of deciding if an AccountsPackage is destined to be a
SnapshotPackage or not (or just used by AccountsHashVerifier).

!!! New behavior changes !!!

Taking snapshots (both bank and archive) **MUST** succeed.

This is required because of how the last full snapshot slot is
calculated, which is used by AccountsBackgroundService when calling
`clean_accounts()`.

File system calls are now unwrapped and will result in a crash. As Trent told me:

>Well I think if a snapshot fails due to some IO error, it's very likely that the operator is going to have to intervene before it works.  We should exit error in this case, otherwise the validator might happily spin for several more hours, never successfully writing a complete snapshot, before something else brings it down.  This would leave the validator's last local snapshot many more slots behind than it would be had we exited outright and potentially force the operator to abandon ledger continuity in favor of a quick catchup

Other errors will set the `exit` flag to `true`, and the node will gracefully shutdown.

Fixes #19167 
Fixes #19168
2021-08-31 18:33:27 -05:00
.buildkite
.github
.travis
account-decoder chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
accounts-bench hash calculation adds really old slots to dirty_stores (#19434) 2021-08-26 14:32:43 -05:00
accounts-cluster-bench Deprecate FeeCalculator returning APIs (#19120) 2021-08-13 09:08:20 -07:00
banking-bench Bank::new -> Bank::new_for_benches (#19063) 2021-08-04 17:30:43 -05:00
banks-client Dedupe SendTransactionService(s) (#19354) 2021-08-23 22:32:15 +00:00
banks-interface chore: bump serde from 1.0.129 to 1.0.130 (#19497) 2021-08-30 18:19:34 +00:00
banks-server Add parameter to allow setting max-retries for SendTransaction rpc (#19387) 2021-08-24 22:44:13 -06:00
bench-streamer chore: cargo +nightly clippy --fix -Z unstable-options 2021-06-18 10:42:46 -07:00
bench-tps chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
book/src/proposals Proposal to handle duplicate blocks (#16127) 2021-07-08 04:36:11 +00:00
ci Remove tar-rs audit ignore (#19344) 2021-08-20 14:34:48 -06:00
clap-utils chore: bump thiserror from 1.0.26 to 1.0.28 (#19494) 2021-08-30 09:35:37 -06:00
cli chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
cli-config chore: bump serde from 1.0.129 to 1.0.130 (#19497) 2021-08-30 18:19:34 +00:00
cli-output chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
client chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
core Make background services aware of incremental snapshots (#19401) 2021-08-31 18:33:27 -05:00
crate-features Bump jsonrpc crates and remove old tokio (#18779) 2021-07-26 12:32:17 -06:00
docs fix typo in calling-between-programs developing documentation (#19502) 2021-08-30 19:17:43 +00:00
dos adds validator flag to allow private ip addresses (#18850) 2021-07-23 15:25:03 +00:00
download-utils change untar to use unpack instead of unpack_in (#19216) 2021-08-18 15:49:02 -05:00
entry chore: bump serde from 1.0.129 to 1.0.130 (#19497) 2021-08-30 18:19:34 +00:00
explorer chore:(deps): bump @types/node from 16.7.6 to 16.7.8 in /explorer (#19519) 2021-08-31 08:06:42 +00:00
faucet chore: bump serde from 1.0.129 to 1.0.130 (#19497) 2021-08-30 18:19:34 +00:00
frozen-abi chore: bump sha2 from 0.9.5 to 0.9.6 (#19511) 2021-08-30 19:58:26 -06:00
genesis chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
genesis-utils chore: cargo +nightly clippy --fix -Z unstable-options 2021-06-18 10:42:46 -07:00
gossip adds logs when push-vote panics with invalid vote-index (#19485) 2021-08-31 12:15:07 +00:00
install chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
keygen chore: cargo +nightly clippy --fix -Z unstable-options 2021-06-18 10:42:46 -07:00
ledger chore: bump futures from 0.3.16 to 0.3.17 (#19513) 2021-08-30 22:12:38 -06:00
ledger-tool Make background services aware of incremental snapshots (#19401) 2021-08-31 18:33:27 -05:00
local-cluster Fix duplicate broadcast test (#19365) 2021-08-27 17:53:24 -07:00
log-analyzer chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
logger chore: bump env_logger from 0.8.4 to 0.9.0 (#18690) 2021-07-15 17:49:59 +00:00
measure Add incremental snapshot utils (#18504) 2021-07-22 14:40:37 -05:00
merkle-root-bench chore: bump log from 0.4.11 to 0.4.14 (#18323) 2021-06-30 18:32:01 +00:00
merkle-tree chore: bump matches from 0.1.8 to 0.1.9 (#19220) 2021-08-13 16:37:45 +00:00
metrics chore: bump env_logger from 0.8.4 to 0.9.0 (#18690) 2021-07-15 17:49:59 +00:00
multinode-demo Preserve pre-existing vote/stake keypairs for bootstrap validator (#19405) 2021-08-24 22:29:55 +00:00
net Preserve pre-existing vote/stake keypairs for bootstrap validator (#19405) 2021-08-24 22:29:55 +00:00
net-shaper chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
net-utils chore: bump serde from 1.0.129 to 1.0.130 (#19497) 2021-08-30 18:19:34 +00:00
notifier chore: bump reqwest from 0.11.2 to 0.11.4 (#18362) 2021-07-01 09:59:30 -06:00
perf chore: bump serde from 1.0.129 to 1.0.130 (#19497) 2021-08-30 18:19:34 +00:00
poh Store versioned transactions in the ledger, disabled by default (#19139) 2021-08-17 15:17:56 -07:00
poh-bench Move entry/poh to own crate to speed up poh bench build (#18225) 2021-07-14 14:16:29 +02:00
program-runtime Add solana-program-runtime crate (#19438) 2021-08-27 00:30:36 +00:00
program-test chore: bump serde from 1.0.129 to 1.0.130 (#19497) 2021-08-30 18:19:34 +00:00
programs Feature cleanup (#19528) 2021-08-31 14:51:26 -07:00
rayon-threadlimit Bump version to v1.8.0 (#17541) 2021-05-27 08:51:53 -07:00
rbpf-cli chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
remote-wallet chore: bump parking_lot from 0.10.2 to 0.11.0 (#18421) 2021-07-05 15:11:35 +00:00
replica-node chore: bump serde from 1.0.129 to 1.0.130 (#19497) 2021-08-30 18:19:34 +00:00
rpc chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
runtime Make background services aware of incremental snapshots (#19401) 2021-08-31 18:33:27 -05:00
scripts chore: bump serde from 1.0.129 to 1.0.130 (#19497) 2021-08-30 18:19:34 +00:00
sdk Feature cleanup (#19528) 2021-08-31 14:51:26 -07:00
send-transaction-service Add parameter to allow setting max-retries for SendTransaction rpc (#19387) 2021-08-24 22:44:13 -06:00
stake-accounts Deprecate FeeCalculator returning APIs (#19120) 2021-08-13 09:08:20 -07:00
storage-bigtable chore: bump futures from 0.3.16 to 0.3.17 (#19513) 2021-08-30 22:12:38 -06:00
storage-proto chore: bump serde from 1.0.129 to 1.0.130 (#19497) 2021-08-30 18:19:34 +00:00
streamer chore: bump libc from 0.2.100 to 0.2.101 (#19442) 2021-08-26 12:38:34 -06:00
sys-tuner chore: bump libc from 0.2.100 to 0.2.101 (#19442) 2021-08-26 12:38:34 -06:00
system-test removes redundant allow-private-addr from system tests 2021-08-13 14:20:27 +00:00
tokens Deprecate FeeCalculator returning APIs (#19120) 2021-08-13 09:08:20 -07:00
transaction-status chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
upload-perf chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
validator chore: bump serde from 1.0.129 to 1.0.130 (#19497) 2021-08-30 18:19:34 +00:00
version chore: bump serde from 1.0.129 to 1.0.130 (#19497) 2021-08-30 18:19:34 +00:00
watchtower Deprecate FeeCalculator returning APIs (#19120) 2021-08-13 09:08:20 -07:00
web3.js chore: bump rpc-websockets from 7.4.12 to 7.4.14 in /web3.js (#19462) 2021-08-27 08:21:37 +00:00
.clippy.toml
.codecov.yml
.gitignore Generate a script to rerun the failed sub-command in cargo-test-bpf (#18582) 2021-07-15 01:02:42 +00:00
.mergify.yml Automatically request review for community PR's (#18737) 2021-07-16 22:07:44 -05:00
.travis.yml Use rustup default profile (#18727) 2021-07-16 19:42:51 +00:00
CONTRIBUTING.md
Cargo.lock chore: bump serde_json from 1.0.66 to 1.0.67 (#19501) 2021-08-31 06:05:25 +00:00
Cargo.toml Dedupe SendTransactionService(s) (#19354) 2021-08-23 22:32:15 +00:00
LICENSE
README.md Update README.md 2021-08-27 09:39:12 -06:00
RELEASE.md
SECURITY.md
cargo
cargo-build-bpf
cargo-test-bpf
fetch-perf-libs.sh
fetch-spl.sh feature: add new token program feature (#18780) 2021-07-21 14:34:25 +02:00
run.sh
test-abi.sh

README.md

Solana

Solana crate Solana documentation Build status codecov

Building

1. Install rustc, cargo and rustfmt.

$ curl https://sh.rustup.rs -sSf | sh
$ source $HOME/.cargo/env
$ rustup component add rustfmt

Please make sure you are always using the latest stable rust version by running:

$ rustup update

On Linux systems you may need to install libssl-dev, pkg-config, zlib1g-dev, etc. On Ubuntu:

$ sudo apt-get update
$ sudo apt-get install libssl-dev libudev-dev pkg-config zlib1g-dev llvm clang make

On Mac M1s, make sure you set up your terminal & homebrew to use Rosetta. You can install it with:

$ softwareupdate --install-rosetta

2. Download the source code.

$ git clone https://github.com/solana-labs/solana.git
$ cd solana

3. Build.

$ cargo build

4. Run a minimal local cluster.

$ ./run.sh

Testing

Run the test suite:

$ cargo test

Starting a local testnet

Start your own testnet locally, instructions are in the online docs.

Accessing the remote development cluster

  • devnet - stable public cluster for development accessible via devnet.solana.com. Runs 24/7. Learn more about the public clusters

Benchmarking

First install the nightly build of rustc. cargo bench requires use of the unstable features only available in the nightly build.

$ rustup install nightly

Run the benchmarks:

$ cargo +nightly bench

Release Process

The release process for this project is described here.

Code coverage

To generate code coverage statistics:

$ scripts/coverage.sh
$ open target/cov/lcov-local/index.html

Why coverage? While most see coverage as a code quality metric, we see it primarily as a developer productivity metric. When a developer makes a change to the codebase, presumably it's a solution to some problem. Our unit-test suite is how we encode the set of problems the codebase solves. Running the test suite should indicate that your change didn't infringe on anyone else's solutions. Adding a test protects your solution from future changes. Say you don't understand why a line of code exists, try deleting it and running the unit-tests. The nearest test failure should tell you what problem was solved by that code. If no test fails, go ahead and submit a Pull Request that asks, "what problem is solved by this code?" On the other hand, if a test does fail and you can think of a better way to solve the same problem, a Pull Request with your solution would most certainly be welcome! Likewise, if rewriting a test can better communicate what code it's protecting, please send us that patch!

Disclaimer

All claims, content, designs, algorithms, estimates, roadmaps, specifications, and performance measurements described in this project are done with the Solana Foundation's ("SF") good faith efforts. It is up to the reader to check and validate their accuracy and truthfulness. Furthermore nothing in this project constitutes a solicitation for investment.

Any content produced by SF or developer resources that SF provides, are for educational and inspiration purposes only. SF does not encourage, induce or sanction the deployment, integration or use of any such applications (including the code comprising the Solana blockchain protocol) in violation of applicable laws or regulations and hereby prohibits any such deployment, integration or use. This includes use of any such applications by the reader (a) in violation of export control or sanctions laws of the United States or any other applicable jurisdiction, (b) if the reader is located in or ordinarily resident in a country or territory subject to comprehensive sanctions administered by the U.S. Office of Foreign Assets Control (OFAC), or (c) if the reader is or is working on behalf of a Specially Designated National (SDN) or a person subject to similar blocking or denied party prohibitions.

The reader should be aware that U.S. export control and sanctions laws prohibit U.S. persons (and other persons that are subject to such laws) from transacting with persons in certain countries and territories or that are on the SDN list. As a project based primarily on open-source software, it is possible that such sanctioned persons may nevertheless bypass prohibitions, obtain the code comprising the Solana blockchain protocol (or other project code or applications) and deploy, integrate, or otherwise use it. Accordingly, there is a risk to individuals that other persons using the Solana blockchain protocol may be sanctioned persons and that transactions with such persons would be a violation of U.S. export controls and sanctions law. This risk applies to individuals, organizations, and other ecosystem participants that deploy, integrate, or use the Solana blockchain protocol code directly (e.g., as a node operator), and individuals that transact on the Solana blockchain through light clients, third party interfaces, and/or wallet software.