blockworks-foundation
/
solana
mirror of https://github.com/blockworks-foundation/solana.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
22,253 Commits 168 Branches 497 Tags 539 MiB
Commit Graph

784 Commits

Author SHA1 Message Date
Brooks Prumo d2974fcefc
Cleanup typos/comments in Bootstrap (#21171) 2021-11-04 08:21:33 -05:00
Brooks Prumo 7508c36209
Bootstrap falls back to getting highest full snapshot (#21124) 2021-11-03 19:23:30 -05:00
dependabot[bot] fc3b1bc62f
chore: bump sysctl from 0.4.2 to 0.4.3 (#21117) 
Bumps [sysctl](https://github.com/johalun/sysctl-rs) from 0.4.2 to 0.4.3.
- [Release notes](https://github.com/johalun/sysctl-rs/releases)
- [Changelog](https://github.com/johalun/sysctl-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/johalun/sysctl-rs/commits)

---
updated-dependencies:
- dependency-name: sysctl
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-02 00:27:22 +00:00
dependabot[bot] ca51ea3e04
chore: bump libc from 0.2.105 to 0.2.106 (#21093) 
* chore: bump libc from 0.2.105 to 0.2.106

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.105 to 0.2.106.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.105...0.2.106)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-11-01 14:17:45 -06:00
Pavel Strakhov 9fabff5129
Parallel notifications in RPC PubSub (#20543) 
* generate rpc notifications in parallel

* Use multithreaded runtime for pubsub

* add metric for time since creation of rpc notification to queue

* measure notification entry processing

* fix: add n_threads config argument

* configure rayon thread pool for rpc notifications

* add config option for pubsub notification threads

* rename metric to created_to_queue_time_us

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* rename metric to notification_entry_processing_time_us

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* use value_of for rpc_pubsub_notification_threads parsing

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* rename threads to sol-sub-notif-N

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* fix rpc tests for TimestampedNotificationEntry

* rustfmt

* use rayon thread limit for rpc

Co-authored-by: Pavel Strakhov <p.strakhov@iconic.vc>
Co-authored-by: Alexander Polakov <a.polakov@zubr.io>
Co-authored-by: Nikita Podoliako <bananaelecitrus@gmail.com>
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
 2021-11-01 00:17:24 -06:00
dependabot[bot] 855b3ccdc0
chore: bump signal-hook from 0.2.3 to 0.3.10 (#21081) 
* chore: bump signal-hook from 0.2.3 to 0.3.10

Bumps [signal-hook](https://github.com/vorner/signal-hook) from 0.2.3 to 0.3.10.
- [Release notes](https://github.com/vorner/signal-hook/releases)
- [Changelog](https://github.com/vorner/signal-hook/blob/master/CHANGELOG.md)
- [Commits](https://github.com/vorner/signal-hook/compare/v0.2.3...v0.3.10)

---
updated-dependencies:
- dependency-name: signal-hook
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update apis

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-10-30 00:21:54 -06:00
Brooks Prumo 9d07746642
Retain trusted peer snapshot hashes bugfix (#21077) 
When retaining trusted peer snapshot hashes, and the peer's full
snapshot hashes match a trusted snapshot hash, but the peer doesn't have
incremental snapshot hashes, that's fine; the peer's hashes should be
retained, not discarded.
 2021-10-29 20:21:33 -05:00
Brooks Prumo 1453fce6c6
Check and unwrap() Option instead of .iter().for_each() (#21072) 2021-10-29 14:17:03 -05:00
sakridge a8d78e89d3
Move test-validator to own module to reduce core dependencies (#20658) 
* Move test-validator to own module to reduce core dependencies

* Fix a few TestValidator paths

* Use solana_test_validator crate for solana_test_validator bin

* Move client int tests to separate crate

Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-10-29 01:27:07 +00:00
dependabot[bot] cb1947bea2
chore: bump libc from 0.2.104 to 0.2.105 (#20972) 
* chore: bump libc from 0.2.104 to 0.2.105

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.104 to 0.2.105.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.104...0.2.105)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-10-26 00:14:58 -06:00
Jeff Biseda 6470560dd1
check OS network limits when starting validator (#20874) 2021-10-25 18:25:55 -07:00
Paul Schaaf 58aa2b964b [solana-test-validator] add support for keypair file parsing for --bpf-program address argument 2021-10-25 16:02:11 -07:00
Jeff Washington (jwash) 43ea579f63
add cli for --accounts-hash-num-passes (#20827) 2021-10-25 09:45:46 -05:00
Michael Vines 350bb561eb Clippy 2021-10-23 08:21:20 +00:00
Michael Vines 86bf071d77 `solana-test-validator --log` now includes version/argument information 2021-10-22 13:01:56 -07:00
Brooks Prumo 4821b0a1d9
Download incremental snapshots during bootstrap (#20696) 2021-10-21 09:14:51 -05:00
dependabot[bot] d1d4d687af
chore: bump console from 0.14.1 to 0.15.0 (#20817) 
* chore: bump console from 0.14.1 to 0.15.0

Bumps [console](https://github.com/mitsuhiko/console) from 0.14.1 to 0.15.0.
- [Release notes](https://github.com/mitsuhiko/console/releases)
- [Changelog](https://github.com/mitsuhiko/console/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/console/compare/0.14.1...v0.15.0)

---
updated-dependencies:
- dependency-name: console
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-10-20 21:43:46 +00:00
Trent Nelson 2744a2128c rpc-send-tx-svc: server-side retry knobs 2021-10-20 13:43:27 -06:00
Trent Nelson fe098b5ddc rpc-send-tx-svc: add with_config constructor 2021-10-20 13:43:27 -06:00
dependabot[bot] 66e431dad5
chore: bump libc from 0.2.103 to 0.2.104 (#20764) 
* chore: bump libc from 0.2.103 to 0.2.104

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.103 to 0.2.104.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.103...0.2.104)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-10-19 22:38:02 -06:00
Brooks Prumo 72e1efb847
Refactor port check in bootstrap (#20793) 2021-10-19 21:35:42 +00:00
Brooks Prumo 7a36bf5095
Add CLI args for no-incremental-snapshot-fetch (#20787) 2021-10-19 13:15:30 -05:00
Brooks Prumo 3f559cc2c9
Move bootstrap code into inner module (#20730) 2021-10-19 09:50:15 -05:00
Brooks Prumo 33f4e79589
Refactor out get_rpc_peers() (#20744) 2021-10-18 14:01:52 -05:00
Jeff Washington (jwash) a8e000a2a6
add filler accounts to bloat validator and predict failure (#20491) 
* add filler accounts to bloat validator and predict failure

* assert no accounts match filler

* cleanup magic numbers

* panic if can't load from snapshot with filler accounts specified

* some renames

* renames

* into_par_iter

* clean filler accts, too
 2021-10-11 12:46:27 -05:00
Lijun Wang d621994fee
Accountsdb stream plugin improvement (#20419) 
Support using connection pooling and use multiple threads to do Postgres db operations. The performance is improved from 1500 RPS to 40,000 RPS measured during validator start.

Support multiple plugins at the same time.
 2021-10-08 20:06:58 -07:00
Tao Zhu 177a375479
Tpu vote 1.7 (#20187) (#20494) 
* Add separate vote processing tpu port

* Add feature to send to tpu vote port

* Add vote rejecting sigverify mode

* use packet.meta.is_simple_vote_tx in place of deserialization

* consolidate code that identifies vote tx atcommon path for cpu and gpu

* new key for feature set

* banking forward tpu vote

* add tpu vote port to dockerfile and other review changes

* Simplify thread id compare

* fix a test; updated cluster_info ABI change

Co-authored-by: Tao Zhu <tao@solana.com>

Co-authored-by: sakridge <sakridge@gmail.com>
 2021-10-07 09:38:23 +00:00
Trent Nelson 767f740305 Bump version to 1.9.0 2021-10-06 17:57:41 -07:00
Justin Starry 129716f3f0
Optimize stakes cache and rewards at epoch boundaries (#20432) 
* Optimize stakes cache and rewards at epoch boundaries

* Fetch from accounts db

* Add cli flag for disabling epoch boundary optimization
 2021-10-06 00:53:26 -04:00
Michael Vines 221343e849 Default --rpc-bind-address to 127.0.0.1 when --private-rpc is provided and --bind-address is not 2021-10-04 16:32:18 -07:00
Brooks Prumo 852b25e167
Move bootstrap code into its own module (#20372) 2021-10-03 13:13:16 -05:00
Michael fc5dd7f3bc
Ignore delinquent stake on exit (#20367) 
* add --ignore-delinquency flag to validator exit and wait-for-restart-window sub commands

* Fix a merge issue

* Add missing variable declaration

* Remove empty line to help CI checks pass

* run rustfmt

* Change argument wording for clarity and verbosity

* Change --ignore-delinquent-stake to --max-delinquent-stake

* cargo fmtgit add validator/src/main.rsgit add validator/src/main.rs

* Adjust per mvines

* Formatting

* Improve input validation

* Please automate cargo fmt somehow
 2021-10-01 19:52:28 +00:00
Lijun Wang fe97cb2ddf
AccountsDb plugin framework (#20047) 
Summary of Changes

Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows

Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.

The code comprises 4 major parts:

accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
 2021-09-30 14:26:17 -07:00
Ivan Mironov 8ae88632cb
Properly enable unprefixed_malloc_on_supported_platforms in tikv-jemallocator (#20351) 
Trivial typo fix.

Fixes: 4bf6d0c4d7 ("adds unprefixed_malloc_on_supported_platforms to jemalloc (#20317)")
 2021-09-30 12:42:15 -06:00
behzad nouri 4bf6d0c4d7
adds unprefixed_malloc_on_supported_platforms to jemalloc (#20317) 
Without this feature jemalloc is used only for Rust code but not for
bundled C/C++ libraries (like rocksdb).
https://github.com/solana-labs/solana/issues/14366#issuecomment-930404992
 2021-09-29 20:41:09 +00:00
Jeff Washington (jwash) 5810568c65
add --accounts-index-path to be used by disk accounts index (#19710) 
* add --accounts-index-paths to be used by disk accounts index

* add multiple and default to help
 2021-09-28 11:07:47 -05:00
dependabot[bot] 9393654af0
chore: bump libc from 0.2.102 to 0.2.103 (#20280) 
* chore: bump libc from 0.2.102 to 0.2.103

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.102 to 0.2.103.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.102...0.2.103)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
 2021-09-28 07:50:46 +00:00
Brooks Prumo 49e85afdcd
Make rpc_bootstrap() aware of local incremental snapshots (#20171) 2021-09-27 13:05:32 -05:00
behzad nouri 2cf081d863
uses tikv_jemallocator::Jemalloc as the global allocator (#20149) 
https://github.com/solana-labs/solana/pull/16346
switched default allocator from jemalloc to system allocator, but that
has shown regressions in form of higher ram usage causing nodes go OOM:
https://discord.com/channels/428295358100013066/439194979856809985/890413193858539580

This commit sets jemalloc as the default allocator.
 2021-09-24 14:26:25 +00:00
Jon Cinque 567f30aa1a
windows: Make solana-test-validator work (#20099) 
* windows: Make solana-test-validator work

The important changes to get this going on Windows:

* ledger lock needs to be done on a file instead of the directory
* IPC service needs to use the Windows pipe naming scheme
* always disable the JIT
* file logging not possible yet because we can't redirect stderr,
but this will change once env_logger fixes the pipe output target!

* Integrate review feedback
 2021-09-22 23:10:35 +02:00
Jeff Washington (jwash) c1d181add5
add --accounts-index-memory-limit-mb (#19269) 2021-09-19 18:00:15 -05:00
Jeff Washington (jwash) 4e038e94fd
add 'drives' to AccountsIndexConfig (#19989) 2021-09-18 09:54:57 -05:00
Pavel Strakhov 65227f44dc
Optimize RPC pubsub for multiple clients with the same subscription (#18943) 
* reimplement rpc pubsub with a broadcast queue

* update tests for new pubsub implementation

* fix: fix review suggestions

* chore(rpc): add additional pubsub metrics

* integrate max subscriptions check into SubscriptionTracker to reduce locking

* separate subscription control from tracker

* limit memory usage of items in pubsub broadcast queue, improve error handling

* add more pubsub metrics

* add final count metrics to pubsub

* add metric for total number of subscriptions

* fix small review suggestions

* remove by_params from SubscriptionTracker and add node_progress_watchers map instead

* add subscription tracker tests

* add metrics for number of pubsub notifications as a counter

* ignore clippy lint in TokenCounter

* fix underflow in token counter

* reduce queue capacity in pubsub tests

* fix(rpc): fix test timeouts

* fix race in account subscription test

* Add RpcSubscriptions::new_for_tests

Co-authored-by: Pavel Strakhov <p.strakhov@iconic.vc>
Co-authored-by: Nikita Podoliako <n.podoliako@zubr.io>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-09-17 13:40:14 -06:00
Jeff Washington (jwash) 99f5684dc4
AcctIdx: handle future config defaults (#19967) 2021-09-17 13:12:06 -05:00
dependabot[bot] ea15c0b40b
chore: bump libc from 0.2.101 to 0.2.102 (#19924) 
* chore: bump libc from 0.2.101 to 0.2.102

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.101 to 0.2.102.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.101...0.2.102)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-09-16 09:45:01 -06:00
Brooks Prumo 79ade5ec68
Add test_incremental_snapshot_download() to local-cluster (#19746) 2021-09-13 21:44:48 -05:00
Jeff Washington (jwash) b57e86abf2
cache account hash info (#19426) 
* cache account hash info

* ledger_path -> accounts_hash_cache_path
 2021-09-13 20:39:26 -05:00
Brooks Prumo 3f6eb96d6e
Call `Bank::update_accounts_hash()` before taking bank snapshot (#19765) 
This is a bugfix.

When `load_frozen_forks()` called `snapshot_bank()`, the accounts hash
was not computed.  So then when a snapshot archive was eventually
created, its hash would be all 1s, which is clearly wrong.  The fix is
to update the bank's accounts hash before taking the bank snapshot.
 2021-09-13 08:27:50 -05:00
Giles Cope c3a610374a unused deps 2021-09-11 14:33:53 -07:00
Brooks Prumo 7aa5f6b833
Add CLI args for incremental snapshots (#19694) 
Add `--incremental-snapshots` flag to enable incremental snapshots.
This will allow setting `--full-snapshot-interval-slots` and
`--incremental-snapshot-interval-slots`.

Also added `--maximum-incremental-snapshots-to-retain`.

Co-authored-by: Michael Vines <mvines@gmail.com>
 2021-09-10 15:59:26 -05:00
Michael Vines a50576ebe6 | 2021-09-09 18:18:21 -07:00
Justin Starry 46fc92f392
Revert "fixup! test-validator: start logging asap (#19655)" (#19740) 
This reverts commit 64a2d7081b.
 2021-09-09 12:12:17 +00:00
steviez 64a2d7081b
fixup! test-validator: start logging asap (#19655) 2021-09-08 02:56:55 -05:00
Jeff Washington (jwash) 456bf15012
AccountsIndexConfig -> AccountsDbConfig (#19687) 2021-09-08 04:30:38 +00:00
Brooks Prumo fe8ba81ce6
Rename to is_valid instead of is_invalid (#19670) 2021-09-07 09:31:54 -05:00
Brooks Prumo 9d9482b9d8
Plumb `maximum_incremental_snapshot_archives_to_retain` (#19640) 2021-09-06 18:01:56 -05:00
Brooks Prumo 5e25ee5ebe
Add maximum_incremental_snapshot_archives_to_retain to SnapshotConfig (#19612) 2021-09-03 20:21:32 +00:00
Brooks Prumo fb1b853b14
Make wait_for_restart_window() aware of Incremental Snapshots (#19587) 
When the validator is waiting to restart, the snapshot slot is checked.
With Incremental Snapshots, the full snapshot interval is going to be
_much_ higher, which would make this function wait for a looooong time.

So, if there's an incremental snapshot slot when checking, use that
slot, otherwise fall back to the full snapshot slot.
 2021-09-03 11:42:22 -05:00
Brooks Prumo 7ab0aec61f
Rename maximum_full_snapshot_archives_to_retain (#19610) 
To prepare for adding maximum_incremental_snapshot_archives_to_retain,
rename the current field in SnapshotConfig.
 2021-09-03 11:28:10 -05:00
Brooks Prumo 8ac94b2cf4
Add Incremental Snapshot support to RPC (#19559) 
#### Problem

There's no way to get incremental snapshot information from RPC.

#### Summary of Changes

- Add new RPC method, `getHighestSnapshotSlot` that returns a `SnapshotSlotInfo`, which contains both the highest full snapshot slot, and the highest incremental snapshot slot _based on_ the full snapshot.
- Deprecate old RPC method, `getSnapshotSlot`
- Update API docs

Fixes #19579
 2021-09-02 15:25:42 -05:00
Lijun Wang 8378e8790f
Accountsdb replication installment 2 (#19325) 
This is the 2nd installment for the AccountsDb replication.

Summary of Changes

The basic google protocol buffer protocol for replicating updated slots and accounts. tonic/tokio is used for transporting the messages.

The basic framework of the client and server for replicating slots and accounts -- the persisting of accounts in the replica-side will be done at the next PR -- right now -- the accounts are streamed to the replica-node and dumped. Replication for information about Bank is also not done in this PR -- to be addressed in the next PR to limit the change size.

Functionality used by both the client and server side are encapsulated in the replica-lib crate.

There is no impact to the existing validator by default.

Tests:

Observe the confirmed slots replicated to the replica-node.
Observe the accounts for the confirmed slot are received at the replica-node side.
 2021-09-01 14:10:16 -07:00
Brooks Prumo 1d5a8ebc6a
Revert "Add LastFullSnapshotSlot to SnapshotConfig (#19341)" (#19529) 
This reverts commit 4d361af976.
 2021-08-31 22:03:19 -05:00
dependabot[bot] 6a53ec28e0
chore: bump serde from 1.0.129 to 1.0.130 (#19497) 
* chore: bump serde from 1.0.129 to 1.0.130

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.129 to 1.0.130.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.129...v1.0.130)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-08-30 18:19:34 +00:00
dependabot[bot] e492638b7d
chore: bump libc from 0.2.100 to 0.2.101 (#19442) 
* chore: bump libc from 0.2.100 to 0.2.101

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.100 to 0.2.101.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.100...0.2.101)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
 2021-08-26 12:38:34 -06:00
Jack May 481ee48c35
Ignore test-ledger when running test-validator (#19433) 2021-08-25 15:48:12 -07:00
dependabot[bot] 46890ac197
chore: bump serde from 1.0.128 to 1.0.129 (#19395) 
* chore: bump serde from 1.0.128 to 1.0.129

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.128 to 1.0.129.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.128...v1.0.129)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-08-24 12:32:30 -06:00
dependabot[bot] c0c95e88d8
chore: bump libc from 0.2.99 to 0.2.100 (#19371) 
* chore: bump libc from 0.2.99 to 0.2.100

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.99 to 0.2.100.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.99...0.2.100)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-08-23 09:34:32 -06:00
dependabot[bot] aea9960c75
chore: bump serde from 1.0.127 to 1.0.128 (#19370) 
* chore: bump serde from 1.0.127 to 1.0.128

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.127 to 1.0.128.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.127...v1.0.128)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
 2021-08-23 09:33:34 -06:00
Brooks Prumo 6d939811e9
Name snapshots consistently (#19346) 
#### Problem

Snapshot names are overloaded, and there are multiple terms that mean the same thing. This is confusing. Here's a list of ones in the codebase that I've found:

```
- snapshot_dir
- snapshots_dir
- snapshot_path
- snapshot_output_dir
- snapshot_package_output_path
- snapshot_archives_dir
```

#### Summary of Changes

For all the ones that are about the directory where snapshot archives are stored, ensure they are `snapshot_archives_dir`. For the ones about the (bank) snapshots directory, set to `bank_snapshots_dir`.


Co-authored-by: Michael Vines <mvines@gmail.com>
 2021-08-21 15:41:03 -05:00
Brooks Prumo 4d361af976
Add LastFullSnapshotSlot to SnapshotConfig (#19341) 2021-08-20 17:06:53 +00:00
Trent Nelson e0bc5fa690 validator: Trusted validators are now called known validators 2021-08-19 22:43:49 -06:00
Jeff Washington (jwash) 7c70f2158b
accounts_index_bins to AccountsIndexConfig (#19257) 
* accounts_index_bins to AccountsIndexConfig

* rename param bins -> config

* rename BINS_FOR* to ACCOUNTS_INDEX_CONFIG_FOR*
 2021-08-17 14:50:01 -05:00
Michael Vines b15fa9fbd2 Add EtcdTowerStorage 2021-08-14 09:46:36 -07:00
Jeff Washington (jwash) e91988c977
cli for num account index bins (#19085) 2021-08-11 11:45:25 -05:00
Michael Vines 7ddda30126 `solana-test-validator` now uses FileTowerStorage 2021-08-11 00:20:46 -07:00
Michael Vines e9722474eb Move tower storage into its own module 2021-08-11 00:20:46 -07:00
dependabot[bot] 536b763751
chore: bump libc from 0.2.98 to 0.2.99 (#19142) 
* chore: bump libc from 0.2.98 to 0.2.99

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.98 to 0.2.99.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.98...0.2.99)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-08-10 10:54:54 -06:00
Brooks Prumo fd937548a0
Move SnapshotArchiveInfo and friends into its own module (#19114) 2021-08-08 07:57:06 -05:00
Brooks Prumo 00890957ee
Add snapshot_utils::bank_from_latest_snapshot_archives() (#18983) 
While reviewing PR #18565, as issue was brought up to refactor some code
around verifying the bank after rebuilding from snapshots.  A new
top-level function has been added to get the latest snapshot archives
and load the bank then verify.  Additionally, new tests have been
written and existing tests have been updated to use this new function.

Fixes #18973

While resolving the issue, it became clear there was some additional
low-hanging fruit this change enabled.  Specifically, the functions
`bank_to_xxx_snapshot_archive()` now return their respective
`SnapshotArchiveInfo`.  And on the flip side,
`bank_from_snapshot_archives()` now takes `SnapshotArchiveInfo`s instead
of separate paths and archive formats.  This bundling simplifies bank
rebuilding.
 2021-08-06 20:16:06 -05:00
Michael Vines 397801a2d8 Extract tower storage details from Tower struct 2021-08-06 10:04:37 -07:00
Jeff Washington (jwash) 3280ae3e9f
add validator option --accounts-db-skip-shrink (#19028) 
* add validator option --accounts-db-skip-shrink

* typo
 2021-08-04 17:28:33 -05:00
Brooks Prumo ca14475085
Add incremental_snapshot_archive_interval_slots to SnapshotConfig (#19026) 
This commit also renames `snapshot_interval_slots` to
`full_snapshot_archive_interval_slots`, updates the comments on the
fields, and make appropriate updates where SnapshotConfig is used.
 2021-08-04 14:40:20 -05:00
dependabot[bot] c16bf02448
chore: bump serde from 1.0.126 to 1.0.127 (#19010) 
* chore: bump serde from 1.0.126 to 1.0.127

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.126 to 1.0.127.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.126...v1.0.127)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-08-02 21:16:34 +00:00
behzad nouri 1cef6fd4b4
allows private addresses by default in test-validator (#18976) 2021-07-29 19:00:23 +00:00
Trent Nelson 71f6d839f9 validator: remove disused cuda config argument 2021-07-29 03:08:52 +00:00
Trent Nelson 8ed0cd0fff validator: check target CPU features earlier 2021-07-29 03:08:52 +00:00
Trent Nelson ed8285c096 validator: start logging asap 2021-07-29 03:08:52 +00:00
Trent Nelson e641f257ef test-validator: move feature check earlier in startup 2021-07-29 03:08:52 +00:00
Trent Nelson ee65ffb3c8 test-validator: start logging asap 2021-07-29 03:08:52 +00:00
Tyera Eulberg 8596db8f53
Bump jsonrpc crates and remove old tokio (#18779) 
* Bump jsonrpc crates and replace old tokio

* Bump tokio

* getBlockTime

* getBlocks

* getBlocksWithLimit, getInflationReward

* getBlock

* getFirstAvailableBlock

* getTransaction

* getSignaturesForAddress

* getSignatureStatuses

* Remove superfluous runtime
 2021-07-26 12:32:17 -06:00
behzad nouri d2d5f36a3c
adds validator flag to allow private ip addresses (#18850) 2021-07-23 15:25:03 +00:00
Brooks Prumo d1debcd971
Add incremental snapshot utils (#18504) 
This commit adds high-level functions for creating and loading-from
incremental snapshots, plus all low-level functions required to perform
those tasks.  This commit **does not** add taking incremental snapshots
as part of a running validator, nor starting up a node with an
incremental snapshot; just laying ground work.

Additionally, `snapshot_utils` and `serde_snapshot` have been
refactored to use a common code paths for the different snapshots.

Also of note, some renaming has happened:
  1. Snapshots are now either `full_` or `incremental_` throughout the
     codebase.  If not specified, the code applies to both.
  2. Bank snapshots now are called "bank snapshots"
     (before they were called "slot snapshots", "bank snapshots", or
      just "snapshots").  The one exception is within `Bank`, where they
     are still just "snapshots", because they are already "bank
     snapshots".
  3. Snapshot archives now have `_archive` in the code.  This
     should clear up an ambiguity between bank snapshots and snapshot
     archives.
 2021-07-22 14:40:37 -05:00
Michael Vines 61865c0ee0 `solana-validator set-identity` now loads the tower file for the new identity 2021-07-21 22:22:08 -07:00
dependabot[bot] 6abafac479
chore: bump fd-lock from 2.0.0 to 3.0.0 (#18756) 
* chore: bump fd-lock from 2.0.0 to 3.0.0

Bumps [fd-lock](https://github.com/yoshuawuyts/fd-lock) from 2.0.0 to 3.0.0.
- [Release notes](https://github.com/yoshuawuyts/fd-lock/releases)
- [Commits](https://github.com/yoshuawuyts/fd-lock/commits)

---
updated-dependencies:
- dependency-name: fd-lock
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Use new api

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-07-19 17:22:11 +00:00
Michael Vines c418e8f370 wait-for-restart-window command now accepts an optional --identity argument 2021-07-14 23:01:14 -07:00
sakridge 7f2254225e
Move entry/poh to own crate to speed up poh bench build (#18225) 2021-07-14 14:16:29 +02:00
dependabot[bot] c853da7424
chore: bump libc from 0.2.97 to 0.2.98 (#18516) 
* chore: bump libc from 0.2.97 to 0.2.98

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.97 to 0.2.98.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.97...0.2.98)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-07-08 17:31:16 +00:00
Michael Vines b6792a3328 Add ability to change the validator identity at runtime 2021-07-01 17:50:04 -07:00
Michael Vines bf157506e8 Remove id ref 2021-07-01 17:50:04 -07:00
Brooks Prumo 45d54b1fc6
Add SnapshotArchiveInfo and refactor functions in snapshot_utils (#18232) 2021-07-01 12:20:56 -05:00
Brooks Prumo 89a3e4f91e
Move SnapshotConfig into its own module (#18331) 
Also move ArchiveFormat to snapshot_utils, and do not
reexport SnapshotVersion.
 2021-07-01 08:55:26 -05:00
dependabot[bot] 78968d132f
chore: bump log from 0.4.11 to 0.4.14 (#18323) 
* chore: bump log from 0.4.11 to 0.4.14

Bumps [log](https://github.com/rust-lang/log) from 0.4.11 to 0.4.14.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/compare/0.4.11...0.4.14)

---
updated-dependencies:
- dependency-name: log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Make version consistent

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-06-30 18:32:01 +00:00
Trent Nelson 02b14caa5f test-validator: hold rent constant with `--slots-per-epoch` 2021-06-30 00:46:12 -06:00
Michael Vines 314102cb54 Remove redundant JsonRpcConfig::identity_pubkey field 2021-06-22 17:20:11 -07:00
behzad nouri 58e115275a obtains shred-version from entrypoint's ip-echo-server in validator-main 2021-06-21 19:37:16 +00:00
behzad nouri 598093b5db adds shred-version to ip-echo-server response 
When starting a validator, the node initially joins gossip with
shred_verison = 0, until it adopts the entrypoint's shred-version:
https://github.com/solana-labs/solana/blob/9b182f408/validator/src/main.rs#L417

Depending on the load on the entrypoint, this adopting entrypoint
shred-version through gossip sometimes becomes very slow, and causes
several problems in gossip because we have to partially support
shred_version == 0 which is a source of leaking crds values from one
cluster to another. e.g. see
https://github.com/solana-labs/solana/pull/17899
and the other linked issues there.

In order to remove shred_version == 0 from gossip, this commit adds
shred-version to ip-echo-server response. Once the entrypoints are
updated, on validator start-up, if --expected_shred_version is not
specified we will obtain shred-version from the entrypoint using
ip-echo-server.
 2021-06-21 19:37:16 +00:00
dependabot[bot] d458fac2ff
chore: bump bincode from 1.3.1 to 1.3.3 (#18087) 
* chore: bump bincode from 1.3.1 to 1.3.3

Bumps [bincode](https://github.com/servo/bincode) from 1.3.1 to 1.3.3.
- [Release notes](https://github.com/servo/bincode/releases)
- [Commits](https://github.com/servo/bincode/compare/v1.3.1...v1.3.3)

---
updated-dependencies:
- dependency-name: bincode
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-06-21 16:06:31 +00:00
dependabot[bot] 491e4450cc
chore: bump libc from 0.2.96 to 0.2.97 (#18060) 
* chore: bump libc from 0.2.96 to 0.2.97

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.96 to 0.2.97.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.96...0.2.97)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-06-18 11:51:06 -06:00
Alexander Meißner 6514096a67 chore: cargo +nightly clippy --fix -Z unstable-options 2021-06-18 10:42:46 -07:00
Trent Nelson 5efc48fc69 validator: expose max active pubsub subscriptions to CLI 2021-06-17 06:32:52 +00:00
Michael Vines f1ebbbab8f Clean up remove_all_authorized_voters() 2021-06-16 15:37:24 -07:00
Lijun Wang 269d995832
Make account shrink configurable #17544 (#17778) 
1. Added both options for measuring space usage using total accounts usage and for individual store shrink ratio using an enum. Validator CLI options: --accounts-shrink-optimize-total-space and --accounts-shrink-ratio
2. Added code for selecting candidates based on total usage in a separate function select_candidates_by_total_usage
3. Added unit tests for the new functions added
4. The default implementations is kept at 0.8 shrink ratio with --accounts-shrink-optimize-total-space set to true

Fixes #17544
 2021-06-09 21:21:32 -07:00
dependabot[bot] 5c72fef85a
chore: bump signal-hook from 0.1.15 to 0.2.3 (#17827) 
Bumps [signal-hook](https://github.com/vorner/signal-hook) from 0.1.15 to 0.2.3.
- [Release notes](https://github.com/vorner/signal-hook/releases)
- [Changelog](https://github.com/vorner/signal-hook/blob/v0.2.3/CHANGELOG.md)
- [Commits](https://github.com/vorner/signal-hook/compare/v0.1.15...v0.2.3)

---
updated-dependencies:
- dependency-name: signal-hook
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-08 12:12:17 -06:00
Govlzkoy e0ab5ee4f8
update dependence version for gag to latest support windows (#17801) 
* update dependence version for gag to leatest support windows

* fix compile on windows

* add Cargo.lock
 2021-06-07 23:17:16 -07:00
Tyera Eulberg 2e998ed11d
Bump jsonrpc crates (#17797) 
* Bump jsonrpc crates

* Update error text

* gitignore for rpc
 2021-06-07 18:03:20 +00:00
Tyera Eulberg 544b3c0d17
Create solana-poh and move remaining rpc modules to solana-rpc (#17698) 
* Create solana-poh crate

* Move BigTableUploadService to solana-ledger

* Add solana-rpc to workspace

* Move dependencies to solana-rpc

* Move remaining rpc modules to solana-rpc

* Single use statement solana-poh

* Single use statement solana-rpc
 2021-06-04 09:23:06 -06:00
Tyera Eulberg 3a647c4bea
Rename ValidatorExit and move to sdk (#17728) 2021-06-04 03:06:13 +00:00
dependabot[bot] a43c29e858
Bump indicatif from 0.15.0 to 0.16.2 (#17628) 
* Bump indicatif from 0.15.0 to 0.16.2

Bumps [indicatif](https://github.com/mitsuhiko/indicatif) from 0.15.0 to 0.16.2.
- [Release notes](https://github.com/mitsuhiko/indicatif/releases)
- [Commits](https://github.com/mitsuhiko/indicatif/compare/0.15.0...0.16.2)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* Fix message types

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-06-01 17:38:02 +00:00
dependabot[bot] ab0f4ff835
Bump serde from 1.0.122 to 1.0.126 (#17618) 
* Bump serde from 1.0.122 to 1.0.126

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.122 to 1.0.126.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.122...v1.0.126)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
 2021-05-31 22:41:25 +00:00
Trent Nelson 974a96738a test-validator: add an arg to control faucet genesis balance 2021-05-28 10:05:37 -06:00
Ryo Onodera 1f97b2365f
Avoid full-range compactions with periodic filtered b.g. ones (#16697) 
* Update rocksdb to v0.16.0

* Promote the infrequent and important log to info!

* Force background compaction by ttl without manual compaction

* Fix test

* Support no compaction mode in test_ledger_cleanup_compaction

* Fix comment

* Make compaction_interval customizable

* Avoid major compaction with periodic filtering...

* Adress lazy_static, special cfs and range check

* Clean up a bit and add comment

* Add comment

* More comments...

* Config code cleanup

* Add comment

* Use .conflicts_with()

* Nullify unneeded delete_range ops for special CFs

* Some clean ups

* Clarify the locking intention

* Ensure special CFs' consistency with PurgeType::CompactionFilter

* Fix comment

* Fix bad copy paste

* Fix various types...

* Don't use tuples

* Add a unit test for compaction_filter

* Fix typo...

* Remove flag and just use new behavior always

* Fix wrong condition negation...

* Doc. about no set_last_purged_slot in purge_slots

* Write a test and fix off-by-one bug....

* Apply suggestions from code review

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* Follow up to github review suggestions

* Fix line-wrapping

* Fix conflict

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
 2021-05-28 16:42:56 +09:00
dependabot[bot] ec1a307a7c
Bump console from 0.11.3 to 0.14.1 (#16301) 
* Bump console from 0.11.3 to 0.14.1

Bumps [console](https://github.com/mitsuhiko/console) from 0.11.3 to 0.14.1.
- [Release notes](https://github.com/mitsuhiko/console/releases)
- [Changelog](https://github.com/mitsuhiko/console/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/console/compare/0.11.3...0.14.1)

Signed-off-by: dependabot[bot] <support@github.com>

* Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-05-27 18:55:27 +00:00
Michael Vines 8eab0e8602
Bump version to v1.8.0 (#17541) 2021-05-27 08:51:53 -07:00
Lijun Wang 54f0fc9f0f
Use type alias for DownloadProgress callback (#17518) 
Convert to use type alias for the callback and cascade the changes to callers. Thanks @jeffwashington for the help making it possible.
Changed the closure for the progress update in the validator main to FnMut and modify the abort count in the closure which is more reliable.
 2021-05-26 13:26:07 -07:00
Tyera Eulberg 9a5330b7eb
Move gossip modules into solana-gossip crate (#17352) 
* Move gossip modules to solana-gossip

* Update Protocol abi digest due to move

* Move gossip benches and hook up CI

* Remove unneeded Result entries

* Single use statements
 2021-05-26 09:15:46 -06:00
Lijun Wang 4c17243157
snapshot download enhancement (#17415) 
1. Allow the validator bootstrap code to specify the minimal snapshot download speed. If the snapshot download speed is detected below that, a different RPC can be retried. The default is 10MB/sec.

2. To prevent spinning on a number of sub-optimal choices and not making progress, the abort/retry logic is implemented with the following safe guards:
2.1 at maximum we do this retry for 5 times -- this number is configurable with default 5.
2.2 if the download in one notification round (5 second) is more than 2%, do not do retry -- it is not as bad anyway.
2.3 if the remaining estimate time is less than 1 minutes, do not abort retry as it will be done quickly anyway.
2.4 We do this abort/retry logic only at the first notification to avoid wasting download efforts -- the reasoning is being opportunistic and too greedy may not achieve overall shorter download time.

3. The download_snapshot and download_file is modified with the option allowing caller to notified of download progress via a callback. This allows the business logic of retrying to the place it belongs.
 2021-05-25 09:32:12 -07:00
Tyera Eulberg 41ec1c8d50
Add blockstore-root-scan for api nodes on boot (#17402) 
* Add blockstore-root-scan for api nodes on boot

* Ensure cluster-confirmed root and parents are set as root in blockstore in load_frozen_forks()

* Plumb rpc-scan-and-fix-roots validator flag
 2021-05-24 13:24:47 -06:00
sakridge a8dca3976b
Refactor genesis download/load/check functions (#17276) 
* Refactor genesis ingest functions

* Consolidate genesis.bin/genesis.tar.bz2 references
 2021-05-24 16:45:36 +02:00
Michael Vines 305d9dd3f4 Validator progress bars are now rendered when stdout is not a terminal 2021-05-19 07:32:39 +00:00
Tyera Eulberg 827355a6b1
Create solana-rpc crate and move subscriptions (#17320) 
* Move non_circulating_supply to runtime

* Add solana-rpc crate and move max_slots

* Move subscriptions to solana-rpc

* Single use statements
 2021-05-19 00:54:28 -06:00
Trent Nelson 2c8dde7224 test-validator: Hint at airdrop when wallet is unavailable 2021-05-14 11:14:27 -06:00
Trent Nelson 754c708473 test-validator: Display more cluster info in dash 2021-05-14 08:36:08 +00:00
Tyera Eulberg 27004f1b76
Return error for excluded secondary-index keys (#17193) 
* Add runtime helpers to check secondary indexes for key

* Add custom rpc error

* Check secondary-index key inclusion in rpc

* Clone complete AccountSecondaryIndexes into rpc to avoid bank query
 2021-05-13 21:04:21 +00:00
Lijun Wang 9c42a89a43
Issue #17008 -- make snapshot archives to hold on to configurable. (#17158) 
* purge_old_snapshot_archives is changed to take an extra argument 'maximum_snapshots_to_retain' to control the max number of latest snapshot archives to retain. Note the oldest snapshot is always retained as before and is not subjected to this new options.
* The validator and ledger-tool executables are modified with a CLI argument --maximum-snapshots-to-retain. And the options are propagated down the call chains. Their corresponding shell scripts were changed accordingly.
* SnapshotConfig is modified to have an extra field for the maximum_snapshots_to_retain
* Unit tests are developed to cover purge_old_snapshot_archives
 2021-05-12 10:32:27 -07:00
Jeff Washington (jwash) 7d96f78821
include/exclude keys on account secondary index (#17110) 
* AccountSecondaryIndexes.include/exclude

* use normal scan if key is not indexed

* add a test to ask for a scan for an excluded secondary index

* fix up cli args
 2021-05-11 22:06:22 +00:00
Michael Vines ec2b06d81d `solana-validator exit` now uses `process::exit()` to ensure prompt termination 2021-05-07 10:33:51 -07:00
Michael Vines 9ba2c53b85 Add --tower argument to specify where tower files are persisted 2021-05-05 12:20:39 -07:00
Trent Nelson f17b80236f test-validator: Plumb --limit-ledger-size 2021-05-04 08:45:24 +00:00
Michael Vines 6004c0abf5 getLeaderSchedule now supports filtered results based on validator identity 2021-04-21 17:59:26 -07:00
Tyera Eulberg 0924c2d070
Add port and gossip options to solana-test-validator (#16696) 2021-04-21 02:40:52 +00:00
Michael Vines f804ce63c2 Add --faucet-port option 2021-04-14 08:57:49 -07:00
Tyera Eulberg 70f3f7e679
Move obsolete rpc endpoints to separate api for removal (#16500) 
* Move obsolete rpc methods to separate api for removal

* Remove obsolete method from docs

* Fix test using obs method
 2021-04-12 20:33:40 -06:00
Michael Vines 05ad979a2d canonicalize authorized voter filepath 2021-04-12 17:52:55 -07:00
Michael Vines 2229b70c4e Add authorized-voter add/remove-all commands 2021-04-12 15:55:28 -07:00
Michael Vines ef30943c5c Fix up App formatting 2021-04-11 22:32:56 -07:00
Michael Vines a679aebc82 wait-for-restart-window works again for unstaked nodes 2021-04-04 16:52:39 +00:00
Michael Vines c8c89dd5f7 wait-for-restart-window now indicates how far away the next restart window is 2021-04-02 22:45:32 -07:00
Michael Vines aac18d7564 Validator monitor now displays the max retransmit slot 2021-03-30 18:58:32 -07:00
sakridge b99ae8f334
Skip leader slots until a vote lands (#15607) 2021-03-25 18:54:51 -07:00
Michael Vines 4ab98fff02 Avoid a panic when --slots-per-epoch is less than MINIMUM_SLOTS_PER_EPOCH 2021-03-18 05:45:21 +00:00
Michael Vines 04c99cf7ea Add --slots-per-epoch argument 2021-03-17 22:56:41 +00:00
Michael Vines 59c19d9fbf Notice the user when the --mint, --bpf-program, or --clone arguments are ignored 2021-03-17 20:04:53 +00:00
DimAn 20b53eb4b4 Revert to snapshots 2 2021-03-17 10:31:01 -07:00
DimAn 0b42379ed7 Revert to snapshots 
Co-authored-by: Michael Vines <mvines@gmail.com>
 2021-03-17 10:31:01 -07:00
DimAn a43b3674c7 add missed suggestion 2021-03-17 10:31:01 -07:00
DimAn cfb01e26dd Apply suggestions from code review 
Co-authored-by: Michael Vines <mvines@gmail.com>
 2021-03-17 10:31:01 -07:00
DimAn 0209d334bd Add option for separate snapshot location 
(cherry picked from commit 6126878f509c69e23480a5ec22b3271e2b16e072)
 2021-03-17 10:31:01 -07:00
Michael Vines a2eb655322 =1.7.0 2021-03-16 07:51:07 +00:00
Michael Vines eab182188a Fix delinquent stake display 2021-03-13 18:58:04 +00:00
Michael Vines 0c9ca5522c Bump version to v1.7.0 2021-03-13 09:01:21 +00:00
Michael Vines aa2b2d6b75 Default --ledger arg to "ledger" for `solana-validator` and `solana-ledger-tool` 2021-03-11 13:09:30 -08:00
Trent Nelson 24d18b3cf2 docs: add docs links for crates published to crates.io 2021-03-11 08:38:18 +00:00
Michael Vines ac8ccee6b8 Add --force and --monitor options to `exit` subcommand 2021-03-11 05:17:31 +00:00
Michael Vines c836cd85c3 Default to the BPF JIT. Use the `solana-validator --no-bpf-jit` flag to disable 2021-03-10 08:37:35 -08:00
Jeff Washington (jwash) 8a3135d17b
Account->AccountSharedData (#15691) 2021-03-09 15:06:07 -06:00
Michael Vines 76ddc93358 The --identity argument is now required 2021-03-06 19:08:37 -08:00
Michael Vines 04d11ca6c6 Slow down `solana-validator monitor` refresh interval when talking to a real node 2021-03-07 00:51:02 +00:00
sakridge d09112fa6d
PoH batch size calibration (#15717) 2021-03-05 16:01:21 -08:00
Michael Vines 66b781eec3 Add 'unknown' health check state 2021-03-05 17:46:50 +00:00
Michael Vines bd13262b42 Add validator startup process reporting before RPC is available 2021-03-05 08:03:36 -08:00
dependabot[bot] e7a4349b0e chore: bump fd-lock from 1.1.1 to 2.0.0 
Bumps [fd-lock](https://github.com/yoshuawuyts/fd-lock) from 1.1.1 to 2.0.0.
- [Release notes](https://github.com/yoshuawuyts/fd-lock/releases)
- [Commits](https://github.com/yoshuawuyts/fd-lock/commits)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-03-04 18:08:53 -08:00
Michael Vines 24ab84936e Break up RPC API into three categories: minimal, full and admin 2021-03-04 16:39:44 -08:00
Jeff Washington (jwash) 22d2bb9a9c
invert default behavior of index hashing (#15689) 2021-03-03 18:12:18 -06:00
Michael Vines d47f1fae40 Check delinquency before restarting 2021-02-26 13:34:04 -08:00
Michael Vines 0dc482e987 Add wait-for-restart-window subcommand 2021-02-26 13:34:04 -08:00
sakridge 05409e51ce
Increase tpu coalescing and add parameter (#15536) 
Should create larger entries on average
 2021-02-26 09:15:45 -08:00
Trent Nelson 7f7370c306 Re-allow clippy::integer_arithmetic at crate-level 2021-02-17 13:55:08 -07:00
Jeff Washington (jwash) ba02452d75
add validator flag no-accounts-db-index-hashing (#15350) 
* add validator flag no_accounts_db_index_hashing

* add validator flag no_accounts_db_index_hashing
 2021-02-16 21:13:48 +00:00
sakridge 5b8f046c67
More configurable rocksdb compaction (#15213) 
rocksdb compaction can cause long stalls, so
make it more configurable to try and reduce those stalls
and also to coordinate between multiple nodes to not induce
stall at the same time.
 2021-02-14 10:16:30 -08:00
Michael Vines f063f02c41 solana-test-validator now uses the BPF JIT by default, `--no-bpf-jit` to disable 2021-02-09 20:24:14 +00:00
Michael Vines f34b8643c7
Add |solana-validator monitor| subcommand (#15118) 2021-02-05 22:39:23 -08:00
Jeff Washington (jwash) 600ff0d915
calculate hash from store instead of index (#15034) 
* calculate hash from store instead of index

* restore update hash in abs
 2021-02-04 09:00:33 -06:00
behzad nouri 0ad063f4e9
adds flag to disable duplicate instance check (#15006) 2021-02-03 16:26:17 +00:00
Tyera Eulberg d978cd1a2a
TestValidator store cpi and logs 2021-02-01 15:35:33 -07:00
Tyera Eulberg cbb8b79a60
Add validator flag to opt in to cpi and logs storage (#14922) 
* Add validator flag to opt in to cpi and logs storage

* Default TestValidator to opt-in; allow using in multinode-demo

* No clone

Co-authored-by: Carl Lin <carl@solana.com>
 2021-02-01 14:00:51 -07:00
Michael Vines 49c908dc50 Add "init" subcommand 2021-02-01 08:12:28 -08:00
Michael Vines 8993ac0c74 Surface faucet start failures to the user of solana-test-validator 2021-01-28 22:35:58 +00:00
sakridge 5bf5a5ec41
Enable accounts caching by default (#14854) 
Co-authored-by: Carl Lin <carl@solana.com>
 2021-01-26 12:06:39 -08:00
Tyera Eulberg ffa5c7dcc8
Deprecate commitment variants (#14797) 
* Deprecate commitment variants

* Add new CommitmentConfig builders

* Add helpers to avoid allowing deprecated variants

* Remove deprecated transaction-status code

* Include new commitment variants in runtime commitment; allow deprecated as long as old variants persist

* Remove deprecated banks code

* Remove deprecated variants in core; allow deprecated in rpc/rpc-subscriptions for now

* Heavier hand with rpc/rpc-subscription commitment

* Remove deprecated variants from local-cluster

* Remove deprecated variants from various tools

* Remove deprecated variants from validator

* Update docs

* Remove deprecated client code

* Add new variants to cli; remove deprecated variants as possible

* Don't send new commitment variants to old clusters

* Retain deprecated method in test_validator_saves_tower

* Fix clippy matches! suggestion for BPF solana-sdk legacy compile test

* Refactor node version check to handle commitment variants and transaction encoding

* Hide deprecated variants from cli help

* Add cli App comments
 2021-01-26 19:23:07 +00:00
Michael Vines bf1943e489 Add solana-test-validator --warp-slot argument 2021-01-22 21:17:02 -08:00
Michael Vines cbb9ac19b9 Add ability to clone accounts from an RPC endpoint 2021-01-22 13:29:36 -08:00
Tyera Eulberg 4964b0fe61
Cli: default to single gossip (#14673) 
* Init cli RpcClient with chosen commitment; default to single_gossip

* Fill in missing client methods

* Cli tests: make RpcClient commitment specific

* Simply rpc_client calls, using configured commitment

* Check validator vote account with single-gossip commitment
 2021-01-19 15:33:03 -07:00
Ryo Onodera dcaa025822
Configure Bigtable's timeout, enabling by default (#14657) 
* Configure bigtable's timeout when read-only

* Review comments

* Apply nits (thanks!)

Co-authored-by: Michael Vines <mvines@gmail.com>

* Timeout in the streamed decoding as well

Co-authored-by: Michael Vines <mvines@gmail.com>
 2021-01-19 13:57:16 +00:00
Michael Vines 5d9dc609b1 Rename RpcNodeUnhealthy error to NodeUnhealthy, generalize `getHealth` RPC error object for the future 2021-01-18 19:57:25 -08:00
Michael Vines 4003f86f04 Add `getSnapshotSlot` RPC method 2021-01-16 19:31:21 +00:00
Michael Vines dacb95083d Add `getHealth` RPC method 2021-01-16 10:10:48 -08:00
Michael Vines 1c2ae15b1d Improve solana-test-validator output 2021-01-14 23:07:24 -08:00
Michael Vines 11daaadc93 Add --rpc-threads argument 2021-01-13 13:34:46 -08:00
carllin 6dfad0652f
Cache account stores, flush from AccountsBackgroundService (#13140) 2021-01-11 17:00:23 -08:00
Michael Vines 4a66e3eddc Use standard tmp-snapshot- file prefix for the "new_state" archive for better cleanup/consistency 2021-01-11 16:51:26 -08:00
Michael Vines 3bd4c38a84 Clarify log message, the remote snapshot might not actually be newer 2021-01-11 11:53:13 -08:00
Michael Vines 7be6770808 Rename CompressionType to ArchiveFormat 2021-01-09 09:07:49 -08:00
carllin 5affd8aa72
Add secondary indexes (#14212) 2020-12-31 18:06:03 -08:00
Michael Vines 0b23abd479 Bind RPC and faucet to 0.0.0.0 2020-12-31 07:55:26 +00:00
Michael Vines 04bf5ce830 Minor help improvements 2020-12-31 07:55:26 +00:00
Trent Nelson fe667db910 validator: Add experimental flag to select PoH pinned core 2020-12-29 19:15:44 -07:00
sakridge 2074e407cd
Add poh speed check and tick speed calibration (#14292) 2020-12-29 09:35:57 -08:00
Michael Vines 9ddd6f08e8 Persist gossip contact info 2020-12-27 20:46:54 -08:00
Michael Vines 87eb924d2a Remove stray dbg 2020-12-24 10:44:39 -08:00
Michael Vines 65dcb3dc81 Add log message for when a local snapshot is too old 2020-12-22 19:39:17 -08:00
Michael Vines ace360ade2 Multiple entrypoint support 2020-12-22 18:35:31 -08:00
sakridge baa9602411
Add shrink paths (#14238) 2020-12-21 21:33:37 -08:00
Michael Vines 8082a2454c Allow multiple --accounts arguments 2020-12-21 07:39:23 +00:00
Ryo Onodera 3c9b853268
Reject invalid --expected-shred-version (#14183) 
* Reject invalid --expected-shred-version

* less code
 2020-12-18 07:01:26 +00:00
sakridge da7d1e2302
Improved Transaction Forwarding (#13944) 
* Forwarding

* Dedupe leaders

* Use consistent commitment for last_valid_slot in rpc send_transaction

* Plumb rpc send_transaction options into solana-validator

* Extend num slots banking-stage holds forwarded txs

Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2020-12-17 15:37:22 -07:00
Michael Vines 8d700c3b94 Use an ephemeral mint address if the client keypair is not available 
Typically this can occur in a CI environment
 2020-12-16 20:43:08 -08:00
Michael Vines aeda8d8b91 Remove stray println 2020-12-15 16:44:31 -08:00
Michael Vines 00c46c528e Add --reset flag to allow for easy ledger reset 2020-12-15 21:51:37 +00:00
Michael Vines f3272db7f7 Prevent multiple test-validators from using the same ledger directory 2020-12-15 21:51:37 +00:00
Michael Vines 1e977ac025 Bump version to v1.6.0 2020-12-15 18:28:04 +00:00
Michael Vines cdd3e7d856
Remove solana-vote-signer (#14099) 2020-12-13 19:12:20 -08:00
Michael Vines 9f2d154588 Rework TestValidator API to be more like ProgramTest 2020-12-13 07:45:55 +00:00
dependabot[bot] 88d950e5a8
chore: bump log from 0.4.8 to 0.4.11 (#13691) 
* chore: bump log from 0.4.8 to 0.4.11

Bumps [log](https://github.com/rust-lang/log) from 0.4.8 to 0.4.11.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/compare/0.4.8...0.4.11)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2020-12-13 13:48:23 +09:00
dependabot[bot] 8df9bcc784
chore: bump libc from 0.2.80 to 0.2.81 (#14092) 
* chore: bump libc from 0.2.80 to 0.2.81

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.80 to 0.2.81.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.80...0.2.81)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2020-12-13 12:13:17 +09:00
Michael Vines bbad3fe501 TestValidator now implements Drop, no need to close() it 2020-12-11 04:17:38 +00:00
Michael Vines 0a9ff1dc9d Initial solana-test-validator command-line program 2020-12-11 04:17:38 +00:00
Michael Vines 6e9dbb4f6e Add --rpc-max-multiple-accounts to override the getMultipleAccounts JSON RPC maximum 2020-12-07 16:31:01 -08:00
Ryo Onodera 2f374df494
Don't show confusing message for blacklist expires (#13983) 2020-12-07 12:47:27 +00:00
Alexander Meißner a706706572
Validator CLI option to enable just-in-time compilation of BPF (#13789) 
* Adds a CLI option to the validator to enable just-in-time compilation of BPF.

* Refactoring to use bpf_loader_program instead of feature_set to pass JIT flag from the validator CLI to the executor.
 2020-12-07 09:49:55 +01:00
Tyera Eulberg 10c81a2448
Remove rpc_banks from validator (#13882) 
* Remove rpc_banks from validator

* Bump abi-digest
 2020-12-02 03:25:09 +00:00
Michael Vines 254790f8c8 Fully clean up temporary snapshot staging directories 2020-11-30 19:40:06 -08:00
Michael Vines aebf12e28d Add --maximum-local-snapshot-age argument 2020-11-28 09:09:55 +00:00
sakridge c1eb350c47
Allow contact debug interval to be adjusted (#13737) 2020-11-20 14:47:37 -08:00
Michael Vines 5d72e52ad0 Disable the PubSub vote subscription by default 
The --rpc-pubsub-enable-vote-subscription flag may be used to enable it.
The current vote subscription is problematic because it emits a
notification for *every* vote, so hundreds a second in a real cluster.
Critically it's also missing information about *who* is voting,
rendering all those notifications practically useless.

Until these two issues can be resolved, the vote subscription is not
much more than a potential DoS vector.
 2020-11-14 12:36:37 -08:00
Michael Vines 328f59ebef --gossip-host may now be specified with --entrypoint 2020-11-13 06:20:15 +00:00
sakridge 7ee0c9e68a
Add non-progress bar download status (#13370) 2020-11-04 10:32:27 -08:00
Justin Starry 8b0242a5d8
Allow nodes to advertise a different rpc address over gossip (#13053) 
* Allow nodes to advertise a different rpc address over gossip

* Feedback
 2020-10-22 03:31:48 +00:00
Ryo Onodera 81489ccb76
Only fetch snapshot if it's newer than local (#12663) 
* Only fetch snapshot if it's newer than local

* Prefer as_ref over clone

* More nits

* Don't wait forwever for newer snapshot
 2020-10-09 15:05:41 +09:00
Michael Vines 11df2e2236 Bump version to v1.5.0 2020-10-08 04:51:36 +00:00
Ryo Onodera aa70dbfc62
Add --no-port-check to validator (#12245) 2020-10-02 20:36:41 +09:00
Michael Vines f41a73d76a Expose validator cli arguments for pubsub buffer tuning 2020-10-01 20:30:40 -07:00
sakridge 68e5a2ef56
Add RPC notify and banking keys debug (#12396) 2020-09-23 18:46:42 -07:00
Michael Vines 65b247a922 Bind to correct RPC addresses 2020-09-20 07:32:25 +00:00
Michael Vines 76dada3118 Use validator_config for RPC address instead of cluster_info for port verification checks 2020-09-20 02:34:20 +00:00
Michael Vines 1a03afccb1 validator/ cleanup 2020-09-19 08:35:26 -07:00
Ryo Onodera cb8661bd49
Persistent tower (#10718) 
* Save/restore Tower

* Avoid unwrap()

* Rebase cleanups

* Forcibly pass test

* Correct reconcilation of votes after validator resume

* d b g

* Add more tests

* fsync and fix test

* Add test

* Fix fmt

* Debug

* Fix tests...

* save

* Clarify error message and code cleaning around it

* Move most of code out of tower save hot codepath

* Proper comment for the lack of fsync on tower

* Clean up

* Clean up

* Simpler type alias

* Manage tower-restored ancestor slots without banks

* Add comment

* Extract long code blocks...

* Add comment

* Simplify returned tuple...

* Tweak too aggresive log

* Fix typo...

* Add test

* Update comment

* Improve test to require non-empty stray restored slots

* Measure tower save and dump all tower contents

* Log adjust and add threshold related assertions

* cleanup adjust

* Properly lower stray restored slots priority...

* Rust fmt

* Fix test....

* Clarify comments a bit and add TowerError::TooNew

* Further clean-up arround TowerError

* Truly create ancestors by excluding last vote slot

* Add comment for stray_restored_slots

* Add comment for stray_restored_slots

* Use BTreeSet

* Consider root_slot into post-replay adjustment

* Tweak logging

* Add test for stray_restored_ancestors

* Reorder some code

* Better names for unit tests

* Add frozen_abi to SavedTower

* Fold long lines

* Tweak stray ancestors and too old slot history

* Re-adjust error conditon of too old slot history

* Test normal ancestors is checked before stray ones

* Fix conflict, update tests, adjust behavior a bit

* Fix test

* Address review comments

* Last touch!

* Immediately after creating cleaning pr

* Revert stray slots

* Revert comment...

* Report error as metrics

* Revert not to panic! and ignore unfixable test...

* Normalize lockouts.root_slot more strictly

* Add comments for panic! and more assertions

* Proper initialize root without vote account

* Clarify code and comments based on review feedback

* Fix rebase

* Further simplify based on assured tower root

* Reorder code for more readability

Co-authored-by: Michael Vines <mvines@gmail.com>
 2020-09-19 14:03:54 +09:00
Michael Vines 98cfe92745 Rework snapshot download logic to be more forgiving when ` --expected-shred-version` is not provided 2020-09-16 20:34:02 +00:00
Ryo Onodera b85e8497b5
Really skip private rpc port reachable checks (#12239) 2020-09-15 16:36:15 +00:00
Michael Vines daae638781 Add --gossip-validator argument 2020-09-14 20:18:27 -07:00
Michael Vines 63a67f415e Add --restricted-repair-only-mode flag 2020-09-14 20:18:27 -07:00
Michael Vines bdf5274d18 Default snapshot compression is now included in --help output 2020-09-11 18:08:34 +00:00
Michael Vines 6f325d4594 `solana-validator --rpc-bind-address` argument now works as expected 2020-09-10 13:36:13 -07:00
Michael Vines 9ade73841f Default snapshot compression to zstd instead of bzip2 for quicker snapshot generation 2020-09-06 22:48:12 -07:00
Michael Vines d8e2038dda Add --enable-bigtable-ledger-upload flag 2020-09-04 16:01:49 -07:00
sakridge 9b9d559312
Increase message_processor logging to error level (#11945) 2020-08-31 13:11:12 -07:00
Michael Vines f8bb93a0f4 Bump RPC banks up 1 port to avoid web3.js wss port conflict 2020-08-31 06:34:15 +00:00
carllin c8d67aa8eb
Add option for repairing only from trusted validators (#11752) 
Co-authored-by: Carl <carl@solana.com>
 2020-08-21 00:35:11 -07:00
Michael Vines ea88bbdc33 Do not delete any ledger when `--limit-ledger-size` is not provided 2020-08-20 16:15:27 +00:00
Greg Fitzgerald bad486823c
Add a client for BankForks (#10728) 
Also:
* Use BanksClient in solana-tokens
 2020-08-07 08:45:17 -06:00
Michael Vines dfae9a9864 Plumb Bigtable ledger storage into the RPC subsystem 2020-08-05 14:27:12 -07:00
Trent Nelson e12ab9d0dd Bump version to 1.4.0 2020-08-05 12:04:15 -06:00
Michael Vines 1d40685b8e Remove support for 1.1.0 version snapshots 2020-08-04 01:01:50 +00:00
sakridge b85538c532
Turn off gossip before downloading snapshot. (#11087) 2020-07-16 17:05:52 -07:00
dependabot-preview[bot] 5055028231 Bump serde_json from 1.0.54 to 1.0.56 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.54 to 1.0.56.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.54...v1.0.56)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2020-07-14 11:13:22 -07:00
dependabot-preview[bot] cee9177ec7 Bump libc from 0.2.71 to 0.2.72 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.71 to 0.2.72.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.71...0.2.72)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2020-07-13 15:00:27 -07:00
sakridge 960f6f6f07
Decouple genesis fetch and snapshot fetch (#10931) 2020-07-06 16:09:12 -07:00
sakridge 58a475b789
Add db recovery methods (#10838) 2020-07-06 12:43:45 -07:00
sakridge 1ffad2d051
More guard rails for restart with ledger procedure (#10853) 
* Add expected_bank_hash required for supermajority
* Print snapshot hash in ledger-tool create-snapshot.
 2020-06-30 12:43:48 -07:00
Kristofer Peterson 6d81eede93
Add CLI options and runtime support for selection of output snapshot version. (#10536) 2020-06-19 14:38:37 +09:00
Greg Fitzgerald 6ee222363e
Move BankForks to solana_runtime (#10637) 
* Move BankForks to solana_runtime

* Update imports
 2020-06-17 15:27:03 +00:00
sakridge 0de6c444d6
Simd poh (#10604) 
* Simd poh

* Fix poh verify bench
 2020-06-16 23:03:26 -07:00
Ryo Onodera a39df7ee5d
Fix udp port check retry and check all udp ports (#10385) 
* Don't start if udp port is really closed

* Fully check all udp ports

* Remove test code.......

* Add tests and adjust impl a bit

* Add comment

* Move comment a bit

* Move a bit

* clean ups
 2020-06-15 07:36:08 +09:00
R. M. Shea 48febf153b
Change documentation introduction to indicate Foundation IP transfer (#10440) 2020-06-13 14:41:05 -07:00
dependabot-preview[bot] a264a9dd0b Bump serde_json from 1.0.53 to 1.0.54 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.53 to 1.0.54.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.53...v1.0.54)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2020-06-09 16:57:46 -07:00
Kristofer Peterson e23340d89e
Clippy cleanup for all targets and nighly rust (also support 1.44.0) (#10445) 
* address warnings from 'rustup run beta cargo clippy --workspace'

minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs

expect some performance improvement AccountsDB::clean_accounts()

* address warnings from 'rustup run beta cargo clippy --workspace --tests'

* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'

* rustfmt

* fix warning stragglers

* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned

* Remove lock around JsonRpcRequestProcessor (#10417)

automerge

* make ancestors parameter optional to avoid forcing construction of empty hash maps

Co-authored-by: Greg Fitzgerald <greg@solana.com>
 2020-06-09 09:38:14 +09:00
dependabot-preview[bot] b787254bd6
Bump console from 0.10.3 to 0.11.3 (#10314) 
Bumps [console](https://github.com/mitsuhiko/console) from 0.10.3 to 0.11.3.
- [Release notes](https://github.com/mitsuhiko/console/releases)
- [Commits](https://github.com/mitsuhiko/console/commits/v0.11.3)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
 2020-05-30 07:47:41 -06:00
Michael Vines eb45aaa6c3
Added --health-check-slot-distance (#10324) 2020-05-29 15:31:52 -07:00
Michael Vines 9eefb2c929 Bump version to 1.3.0 2020-05-28 15:01:13 -07:00
Michael Vines f60b101920
Verify TPU and serve repair ports are reachable (#10291) 2020-05-28 08:41:06 -07:00
dependabot-preview[bot] 1bfc4c1489
Bump libc from 0.2.70 to 0.2.71 (#10249) 
* Bump libc from 0.2.70 to 0.2.71

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.70 to 0.2.71.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.70...0.2.71)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
 2020-05-26 11:04:15 -07:00
Michael Vines 269db1710e
Retry a couple times before declaring a UDP port unreachable (#10181) 2020-05-22 14:33:01 -07:00
dependabot-preview[bot] 6a6c5f196a
Bump libc from 0.2.69 to 0.2.70 (#10006) 
* Bump libc from 0.2.69 to 0.2.70

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.69 to 0.2.70.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.69...0.2.70)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2020-05-15 06:18:43 -06:00
Jack May eb1acaf927
Remove archiver and storage program (#9992) 
automerge
 2020-05-14 18:22:47 -07:00
Michael Vines 4e4a21f9b7
`solana-gossip spy` can now specify a shred version (#10040) 2020-05-13 19:37:40 -07:00
Michael Vines a75086287c
Use CommitmentConfig::root() when checking accounts, CommitmentConfig::max() may not be available yet (#9999) 
automerge
 2020-05-12 00:24:04 -07:00
Michael Vines 2521f75c18
Advertise node software version in gossip (#9981) 
* Advertise node version in gossip

* Remove solana_clap_utils::version! macro
 2020-05-11 15:02:01 -07:00
dependabot-preview[bot] 6660e93c39
Bump clap from 2.33.0 to 2.33.1 (#9978) 
* Bump clap from 2.33.0 to 2.33.1

Bumps [clap](https://github.com/clap-rs/clap) from 2.33.0 to 2.33.1.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/v2.33.1/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v2.33.0...v2.33.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2020-05-11 14:32:50 -07:00
dependabot-preview[bot] 4fd7526852
Bump signal-hook from 0.1.14 to 0.1.15 (#9979) 
Bumps [signal-hook](https://github.com/vorner/signal-hook) from 0.1.14 to 0.1.15.
- [Release notes](https://github.com/vorner/signal-hook/releases)
- [Changelog](https://github.com/vorner/signal-hook/blob/master/CHANGELOG.md)
- [Commits](https://github.com/vorner/signal-hook/compare/v0.1.14...v0.1.15)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
 2020-05-11 14:32:26 -07:00
dependabot-preview[bot] 0541431ea8
Bump serde_json from 1.0.52 to 1.0.53 (#9957) 
* Bump serde_json from 1.0.52 to 1.0.53

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.52 to 1.0.53.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.52...v1.0.53)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2020-05-10 11:46:29 -07:00
sakridge f98bfda6f9
Security changes (#9923) 
* Move test-only functions to test modules

* Remove sigverify disable

* Remove chacha CTR code
 2020-05-08 10:00:23 -07:00
dependabot-preview[bot] 3757754c89
Bump signal-hook from 0.1.13 to 0.1.14 (#9913) 
Bumps [signal-hook](https://github.com/vorner/signal-hook) from 0.1.13 to 0.1.14.
- [Release notes](https://github.com/vorner/signal-hook/releases)
- [Changelog](https://github.com/vorner/signal-hook/blob/master/CHANGELOG.md)
- [Commits](https://github.com/vorner/signal-hook/compare/v0.1.13...v0.1.14)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
 2020-05-07 08:33:58 -07:00
Ryo Onodera a91236012d
Pass around --max-genesis-archive-unpacked-size (#9161) 
automerge
 2020-04-29 18:53:34 -07:00
Stephen Akridge eb24f3df84 Update dalek version 2020-04-28 21:02:47 -06:00
Michael Vines d67ad70443 Revert "Bump console from 0.10.1 to 0.11.2 (#9763)" 
This reverts commit 2de999fb61.
 2020-04-28 10:43:18 -07:00
dependabot-preview[bot] 2de999fb61
Bump console from 0.10.1 to 0.11.2 (#9763) 
Bumps [console](https://github.com/mitsuhiko/console) from 0.10.1 to 0.11.2.
- [Release notes](https://github.com/mitsuhiko/console/releases)
- [Commits](https://github.com/mitsuhiko/console/compare/0.10.1...0.11.2)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
 2020-04-28 09:36:07 -07:00
dependabot-preview[bot] c5a7df9221
Bump console from 0.10.0 to 0.10.1 (#9726) 
Bumps [console](https://github.com/mitsuhiko/console) from 0.10.0 to 0.10.1.
- [Release notes](https://github.com/mitsuhiko/console/releases)
- [Commits](https://github.com/mitsuhiko/console/compare/0.10.0...0.10.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
 2020-04-27 09:16:24 -06:00
Michael Vines 50f1ec0374
Add support for log rotation, sending SIGUSR1 will cause the log file to be re-opened (#9713) 2020-04-24 14:26:53 -07:00
carllin bab3502260
Push down cluster_info lock (#9594) 
* Push down cluster_info lock

* Rework budget decrement

Co-authored-by: Carl <carl@solana.com>
 2020-04-21 12:54:45 -07:00
Michael Vines 47ae57610a Only build x86_64-unknown-linux-gnu on docs.rs 2020-04-16 19:06:17 -07:00
sakridge 66abe45ea1
Decouple accounts hash calculation from snapshot hash (#9507) 2020-04-16 15:12:20 -07:00
Michael Vines 4ac15e68cf Default to RUST_BACKTRACE=1 for more informative validator logs 2020-04-15 22:37:22 -07:00
Michael Vines 83a96c557d Ensure --dynamic-port-range is wide enough 2020-04-14 12:21:05 -07:00
Michael Vines 37b048effb Improve address in use error message for RPC pubsub 2020-04-13 12:13:37 -07:00
sakridge be77bdef12
Allow lower shred count (#9410) 2020-04-09 16:36:44 -07:00
dependabot-preview[bot] 62cb2cd13c Bump serde_json from 1.0.49 to 1.0.51 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.49 to 1.0.51.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.49...v1.0.51)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2020-04-06 08:40:55 -07:00
sakridge 9493de4443
Add snapshot compression option (#9276) 2020-04-03 13:13:49 -07:00
sakridge a61ddb6f61
max_ledger_slots -> max_ledger_shreds (#9198) 
automerge
 2020-03-31 17:21:19 -07:00
Michael Vines 0e2722c638
solana-validator now supports multiple --authorized-voter arguments (#9174) 
* Use Epoch type

* Vote account's authorized voter is now supported without a validator restart
 2020-03-31 08:23:42 -07:00
sakridge 73e99cc513
Ledger cleanup fixes (#9131) 
* Fix purging happening every slot when cleanup service is not started at slot 0
* Purge by shred count instead of slots since slots can have variable
number of shreds
 2020-03-30 19:02:12 -07:00
dependabot-preview[bot] 2ed3e2160d
Bump serde_json from 1.0.48 to 1.0.49 (#9139) 
* Bump serde_json from 1.0.48 to 1.0.49

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.48 to 1.0.49.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.48...v1.0.49)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Update serde_json in programs

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Justin Starry <justin@solana.com>
 2020-03-28 12:19:40 -07:00
Michael Vines d7fa40087c
Bump version to 1.2.0 2020-03-28 09:44:13 -07:00
Ryo Onodera f987c18a7e
Strictly validate the contents of snapshot/genesis (#8959) 
automerge
 2020-03-25 02:46:41 -07:00
Tyera Eulberg 1b8f9e75dd
Update getSignatureStatus: support multiple signatures, include slot in each response item (#9022) 
* Rename enable-rpc-get-confirmed-block

* Rename RpcTransactionStatus -> RpcTransactionStatusMeta

* Return simplified RpcTransactionStatus; Add support for multiple transactions

* Update docs

* typo
 2020-03-23 11:25:39 -06:00
sakridge 4d2b83d01f
Add option to disable rocks compaction (#9011) 2020-03-23 08:42:32 -07:00
Michael Vines ca791a0378
Ensure --identity is provided when --vote-account is provided (#9014) 
automerge
 2020-03-22 22:21:00 -07:00
Michael Vines 88ba8439fc
Add frozen account support (#8989) 
automerge
 2020-03-22 11:10:04 -07:00
Michael Vines aa24181a53
Remove blockstream unix socket support. RPC or bust (#9004) 
automerge
 2020-03-21 20:17:11 -07:00
sakridge dc347dd3d7
Add Accounts hash consistency halting (#8772) 
* Accounts hash consistency halting

* Add option to inject account hash faults for testing.

Enable option in local cluster test to see that node halts.
 2020-03-16 08:37:31 -07:00
Michael Vines 49706172f3 Quietly re-introduce legacy --voting-keypair/--identity-keypair args for v1.0.6 compatibility 2020-03-15 20:00:58 -07:00
Michael Vines 5481d1a039 Validators now run a full gossip node while looking for a snapshot 2020-03-15 09:31:55 -07:00
Michael Vines 29fb79382c Rework validator vote account defaults to half voting fees 2020-03-13 20:13:33 -07:00
dependabot-preview[bot] 215650f6e7
Bump console from 0.9.2 to 0.10.0 (#8786) 
Bumps [console](https://github.com/mitsuhiko/console) from 0.9.2 to 0.10.0.
- [Release notes](https://github.com/mitsuhiko/console/releases)
- [Commits](https://github.com/mitsuhiko/console/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
 2020-03-11 12:30:15 -06:00
dependabot-preview[bot] e41ff2df66
Bump chrono from 0.4.10 to 0.4.11 (#8755) 
Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.10 to 0.4.11.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/master/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.10...v0.4.11)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
 2020-03-10 08:43:18 -06:00
dependabot-preview[bot] fad08a19cc
Bump serde_json from 1.0.46 to 1.0.48 (#8260) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.46 to 1.0.48.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.46...v1.0.48)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
 2020-03-09 20:20:19 -06:00
Michael Vines 88b1383eed
Permit --no-untrusted-rpc without any --trusted-validators 2020-03-08 22:34:04 -07:00
sakridge 97986a5241
Move download code to download-utils crate (#8704) 2020-03-07 07:08:01 -08:00
Tyera Eulberg a7d1346d51
Remove ask-seed-phrase arg from validator, archiver (#8697) 
* Remove ask-seed-phrase from validator

* Update paper-wallet docs

* Remove ask-seed-phrase from archiver

* Remove unused structs, methods
 2020-03-06 22:22:23 -07:00
Michael Vines cea8067219
Disable setLogFilter RPC API by default (#8693) 
automerge
 2020-03-06 16:03:10 -08:00
Michael Vines 448b957a13
Add --bind-address and --rpc-bind-address validator arguments (#8628) 2020-03-04 22:46:43 -07:00
Grimes f89c22b5ee
`solana catchup` now detects when you try to catchup to yourself (#8635) 
automerge
 2020-03-04 14:44:21 -08:00
Michael Vines 13551885c2 --wait-for-supermajority now requires a SLOT 2020-03-02 12:59:35 -07:00
Michael Vines d677e83ed4 Add ---no-untrusted-rpc flag 2020-03-02 11:49:38 -07:00
Michael Vines b4362cc18b Log RPC node root slot 2020-02-29 09:21:52 -07:00
Michael Vines 0f31adeafb GET for /snapshot.tar.bz2 now redirects to the latest snapshot 2020-02-28 23:23:59 -07:00
Michael Vines 74da2de3b7
Ensure the validator's identity pubkey is not provided as a --trusted-validator (#8525) 
automerge
 2020-02-27 20:26:53 -08:00
Michael Vines fcabc6f799 Rename snapshot.tar.bz2 to snapshot-<slot>-<hash>.tar.bz2 2020-02-26 23:32:53 -07:00
Michael Vines 27c5ec0149 Use the same reqwest features across the repo 2020-02-26 20:47:43 -07:00
Michael Vines d5a7867087 Validate the genesis config downloaded over RPC before accepting it 2020-02-26 14:21:37 -07:00
Michael Vines fbf78b83c4 Add retry mechanism when downloading genesis and snapshots 2020-02-26 14:21:37 -07:00
Michael Vines 0fe74e95fe
Add --no-check-vote-account argument (#8430) 
automerge
 2020-02-24 22:54:51 -08:00
carllin 39282be486
Determine vote_state ahead of time (#8303) 
automerge
 2020-02-24 19:27:04 -08:00
Michael Vines 5c5a06198c Refactor 2020-02-24 14:45:32 -07:00
Michael Vines 394933e53c Fix up trusted validator snapshot selection 2020-02-24 14:45:32 -07:00
Dan Albert 2d9d2f1e99
Update cargo versions from 1.0 to 1.1 (#8397) 2020-02-21 23:09:45 -08:00
sakridge b7386f9d84
Add --trusted-validator support for snapshot hash validation (#8390) 2020-02-21 18:42:24 -08:00
Michael Vines 48f58a88bc Bump version to 1.0.0 2020-02-20 23:52:19 -07:00
Michael Vines cc7c6c960e Search for the validator with the highest snapshot 2020-02-20 17:04:48 -07:00
Tyera Eulberg ab361a8073
Rename KeypairUtil to Signer (#8360) 
automerge
 2020-02-20 13:28:55 -08:00
Michael Vines c4fd81fc1c The getConfirmedBlock RPC API is now disabled by default 
The --enable-rpc-get-confirmed-block flag allows validators to opt-in to
the higher disk usage and IOPS.
 2020-02-11 22:24:08 -07:00
carllin d3712dd26d Factor repair from gossip (#8044) 2020-02-11 13:11:48 -07:00