Go to file
François Garillot f016c9a669
Maintenance : simplify a few patterns, remove unneeded dependencies (#8137)
* Simplify a few pattern matches

* Removing unneeded dependencies, upgrading internal version #s

 net-shaper: Removing log, semver, serde_derive
 bench-tps: Removing serde, serde_derive
 banking-bench: Removing solana
 ledger-tool: Removing bincode, serde, serde_derive
 librapay: Removing solana, language_e2e_tests
 log-analyzer: Removing log, semver, serde_derive
 exchange: Removing solana
 core: Removing crc, memmap, symlink, untrusted
 perf: Removing serde_derive
 genesis: Removing hex, serde_derive
 sdk-c: Removing sha2
 sys-tuner: Removing semver
 bench-exchange: Removing bincode, bs58, env_logger, serde, serde_derive, untrusted, ws
 btc_spv_bin: Removing serde_json
 btc_spv: Removing chrono
 bpf_loader: Removing serde
 ledger: Removing dlopen, dlopen_derive, serde_derive
 move_loader: Removing byteorder, libc, language_e2e_tests
 ownable: Removing serde, serde_derive
 client: Removing rand
 archiver-utils: Removing rand_chacha
 validator: Removing serde_json, tempfile
 param_passing_dep: Removing solana
 failure: Removing log
 vest: Removing log
 vote-signer: Removing bs58, serde
 local-cluster: Removing symlink
 keygen: Removing rpassword
 install: Removing bs58, log
 upload-perf: Removing log
 runtime: Removing serde_json
 stake: Removing rand

* Add modified Cargo.lock

* fixup! Simplify a few pattern matches

* fixup! Simplify a few pattern matches
2020-02-06 10:02:38 -07:00
.buildkite Update 2019-10-25 07:51:12 -07:00
.github Mark PRs stale after 7 days instead of 30 (#7528) 2019-12-17 14:37:35 -07:00
archiver Bump console from 0.9.1 to 0.9.2 2020-01-29 23:22:44 -07:00
archiver-lib Bump serde_json from 1.0.44 to 1.0.46 (#8087) 2020-02-04 16:59:23 -08:00
archiver-utils Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
banking-bench Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
bench-exchange Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
bench-streamer Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
bench-tps Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
book Delete uptime command, report total credits in `solana validators` instead 2020-02-03 16:55:33 -07:00
chacha Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
chacha-cuda Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
chacha-sys Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
ci Broaden pattern in nits.sh to be less fragile (#8090) 2020-02-04 08:53:01 +09:00
clap-utils Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
cli Rename program_utils.rs (#8127) 2020-02-05 12:48:30 -08:00
cli-config Use solana-cli config keypair in solana-keygen (#8074) 2020-01-31 19:27:37 -07:00
client Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
core Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
crate-features Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
faucet Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
genesis Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
genesis-programs Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
gossip Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
install Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
keygen Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
ledger Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
ledger-tool Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
local-cluster Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
log-analyzer Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
logger Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
measure Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
merkle-tree Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
metrics Bump sys-info from 0.5.8 to 0.5.9 (#8089) 2020-02-03 14:49:39 -07:00
multinode-demo Fix cli call to stakes (#7946) 2020-01-23 14:25:40 -08:00
net s/mint/faucet 2020-01-31 12:14:53 -07:00
net-shaper Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
net-utils Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
perf Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
programs Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
rayon-threadlimit Bump sys-info from 0.5.8 to 0.5.9 (#8089) 2020-02-03 14:49:39 -07:00
runtime Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
scripts Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
sdk Rename program_utils.rs (#8127) 2020-02-05 12:48:30 -08:00
sdk-c Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
sys-tuner Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
system-test Add new colo test cases using reduced node count (#8078) 2020-01-31 18:02:48 -08:00
upload-perf Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
validator Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
vote-signer Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
watchtower Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
.appveyor.yml Ignore release branches that exist only for gitbook 2019-09-17 15:31:13 -07:00
.clippy.toml Fix all remaining clippy warnings 2018-07-12 09:40:40 -06:00
.codecov.yml Adjust settings 2018-12-20 12:32:25 -08:00
.gitbook.yaml Update Gitbook YAML to add summary 2019-09-17 20:50:15 -06:00
.gitignore Cleanup book build (#7254) 2019-12-17 14:29:46 -08:00
.mergify.yml Update backport labels 2020-01-17 21:38:39 -07:00
.travis.yml Assume stable is already installed 2019-10-02 23:35:10 -07:00
CONTRIBUTING.md Add guidance for getting PRs merged (#7433) 2019-12-16 12:54:09 -07:00
Cargo.lock Maintenance : simplify a few patterns, remove unneeded dependencies (#8137) 2020-02-06 10:02:38 -07:00
Cargo.toml Use solana-cli config keypair in solana-keygen (#8074) 2020-01-31 19:27:37 -07:00
LICENSE Update LICENSE 2018-06-22 22:41:04 -07:00
README.md Upgrade to rust 1.39.0 (#6939) 2019-11-14 12:27:01 -07:00
RELEASE.md Remove old book location (#7621) 2019-12-23 22:56:42 -07:00
fetch-perf-libs.sh Bump perf libs to v0.18.0 for CUDA 10.2 support 2020-01-24 21:39:49 -07:00
run.sh don't put accounts in a weird location, use the defaults (#7921) 2020-01-22 10:57:37 -08:00

README.md

Solana crate Solana documentation Build status codecov

Blockchain Rebuilt for Scale

Solana™ is a new blockchain architecture built from the ground up for scale. The architecture supports up to 710 thousand transactions per second on a gigabit network.

Disclaimer

All claims, content, designs, algorithms, estimates, roadmaps, specifications, and performance measurements described in this project are done with the author's best effort. It is up to the reader to check and validate their accuracy and truthfulness. Furthermore nothing in this project constitutes a solicitation for investment.

Introduction

It's possible for a centralized database to process 710,000 transactions per second on a standard gigabit network if the transactions are, on average, no more than 176 bytes. A centralized database can also replicate itself and maintain high availability without significantly compromising that transaction rate using the distributed system technique known as Optimistic Concurrency Control [H.T.Kung, J.T.Robinson (1981)]. At Solana, we're demonstrating that these same theoretical limits apply just as well to blockchain on an adversarial network. The key ingredient? Finding a way to share time when nodes can't trust one-another. Once nodes can trust time, suddenly ~40 years of distributed systems research becomes applicable to blockchain!

Perhaps the most striking difference between algorithms obtained by our method and ones based upon timeout is that using timeout produces a traditional distributed algorithm in which the processes operate asynchronously, while our method produces a globally synchronous one in which every process does the same thing at (approximately) the same time. Our method seems to contradict the whole purpose of distributed processing, which is to permit different processes to operate independently and perform different functions. However, if a distributed system is really a single system, then the processes must be synchronized in some way. Conceptually, the easiest way to synchronize processes is to get them all to do the same thing at the same time. Therefore, our method is used to implement a kernel that performs the necessary synchronization--for example, making sure that two different processes do not try to modify a file at the same time. Processes might spend only a small fraction of their time executing the synchronizing kernel; the rest of the time, they can operate independently--e.g., accessing different files. This is an approach we have advocated even when fault-tolerance is not required. The method's basic simplicity makes it easier to understand the precise properties of a system, which is crucial if one is to know just how fault-tolerant the system is. [L.Lamport (1984)]

Furthermore, and much to our surprise, it can be implemented using a mechanism that has existed in Bitcoin since day one. The Bitcoin feature is called nLocktime and it can be used to postdate transactions using block height instead of a timestamp. As a Bitcoin client, you'd use block height instead of a timestamp if you don't trust the network. Block height turns out to be an instance of what's being called a Verifiable Delay Function in cryptography circles. It's a cryptographically secure way to say time has passed. In Solana, we use a far more granular verifiable delay function, a SHA 256 hash chain, to checkpoint the ledger and coordinate consensus. With it, we implement Optimistic Concurrency Control and are now well en route towards that theoretical limit of 710,000 transactions per second.

Architecture

Before you jump into the code, review the online book Solana: Blockchain Rebuilt for Scale.

(The latest development version of the online book is also available here.)

Release Binaries

Official release binaries are available at Github Releases.

Additionally we provide pre-release binaries for the latest code on the edge and beta channels. Note that these pre-release binaries may be less stable than an official release.

Edge channel

Linux (x86_64-unknown-linux-gnu)

mac OS (x86_64-apple-darwin)

Windows (x86_64-pc-windows-msvc)

All platforms

Beta channel

Linux (x86_64-unknown-linux-gnu)

mac OS (x86_64-apple-darwin)

Windows (x86_64-pc-windows-msvc)

All platforms

Developing

Building

Install rustc, cargo and rustfmt:

$ curl https://sh.rustup.rs -sSf | sh
$ source $HOME/.cargo/env
$ rustup component add rustfmt

If your rustc version is lower than 1.39.0, please update it:

$ rustup update

On Linux systems you may need to install libssl-dev, pkg-config, zlib1g-dev, etc. On Ubuntu:

$ sudo apt-get install libssl-dev pkg-config zlib1g-dev llvm clang

Download the source code:

$ git clone https://github.com/solana-labs/solana.git
$ cd solana

Build

$ cargo build

Then to run a minimal local cluster

$ ./run.sh

Testing

Run the test suite:

$ cargo test

Local Testnet

Start your own testnet locally, instructions are in the book Solana: Blockchain Rebuild for Scale: Getting Started.

Remote Testnets

We maintain several testnets:

  • testnet - public stable testnet accessible via testnet.solana.com. Runs 24/7
  • testnet-beta - public beta channel testnet accessible via beta.testnet.solana.com. Runs 24/7
  • testnet-edge - public edge channel testnet accessible via edge.testnet.solana.com. Runs 24/7

Deploy process

They are deployed with the ci/testnet-manager.sh script through a list of scheduled buildkite jobs. Each testnet can be manually manipulated from buildkite as well.

How do I reset the testnet?

Manually trigger the testnet-management pipeline and when prompted select the desired testnet

How can I scale the tx generation rate?

Increase the TX rate by increasing the number of cores on the client machine which is running bench-tps or run multiple clients. Decrease by lowering cores or using the rayon env variable RAYON_NUM_THREADS=<xx>

How can I test a change on the testnet?

Currently, a merged PR is the only way to test a change on the testnet. But you can run your own testnet using the scripts in the net/ directory.

Adjusting the number of clients or validators on the testnet

Edit ci/testnet-manager.sh

Metrics Server Maintenance

Sometimes the dashboard becomes unresponsive. This happens due to glitch in the metrics server. The current solution is to reset the metrics server. Use the following steps.

  1. The server is hosted in a GCP VM instance. Check if the VM instance is down by trying to SSH into it from the GCP console. The name of the VM is metrics-solana-com.
  2. If the VM is inaccessible, reset it from the GCP console.
  3. Once VM is up (or, was already up), the metrics services can be restarted from build automation.
    1. Navigate to https://buildkite.com/solana-labs/metrics-dot-solana-dot-com in your web browser
    2. Click on New Build
    3. This will show a pop up dialog. Click on options drop down.
    4. Type in FORCE_START=true in Environment Variables text box.
    5. Click Create Build
    6. This will restart the metrics services, and the dashboards should be accessible afterwards.

Debugging Testnet

Testnet may exhibit different symptoms of failures. Primary statistics to check are

  1. Rise in Confirmation Time
  2. Nodes are not voting
  3. Panics, and OOM notifications

Check the following if there are any signs of failure.

  1. Did testnet deployment fail?
    1. View buildkite logs for the last deployment: https://buildkite.com/solana-labs/testnet-management
    2. Use the relevant branch
    3. If the deployment failed, look at the build logs. The build artifacts for each remote node is uploaded. It's a good first step to triage from these logs.
  2. You may have to log into remote node if the deployment succeeded, but something failed during runtime.
    1. Get the private key for the testnet deployment from metrics-solana-com GCP instance.
    2. SSH into metrics-solana-com using GCP console and do the following.
    sudo bash
    cd ~buildkite-agent/.ssh
    ls
    
    1. Copy the relevant private key to your local machine
    2. Find the public IP address of the AWS instance for the remote node using AWS console
    3. ssh -i <private key file> ubuntu@<ip address of remote node>
    4. The logs are in ~solana\solana folder

Benchmarking

First install the nightly build of rustc. cargo bench requires use of the unstable features only available in the nightly build.

$ rustup install nightly

Run the benchmarks:

$ cargo +nightly bench

Release Process

The release process for this project is described here.

Code coverage

To generate code coverage statistics:

$ scripts/coverage.sh
$ open target/cov/lcov-local/index.html

Why coverage? While most see coverage as a code quality metric, we see it primarily as a developer productivity metric. When a developer makes a change to the codebase, presumably it's a solution to some problem. Our unit-test suite is how we encode the set of problems the codebase solves. Running the test suite should indicate that your change didn't infringe on anyone else's solutions. Adding a test protects your solution from future changes. Say you don't understand why a line of code exists, try deleting it and running the unit-tests. The nearest test failure should tell you what problem was solved by that code. If no test fails, go ahead and submit a Pull Request that asks, "what problem is solved by this code?" On the other hand, if a test does fail and you can think of a better way to solve the same problem, a Pull Request with your solution would most certainly be welcome! Likewise, if rewriting a test can better communicate what code it's protecting, please send us that patch!