c3a610374a
unused deps
2021-09-11 14:33:53 -07:00
9d9482b9d8
Plumb `maximum_incremental_snapshot_archives_to_retain` ( #19640 )
2021-09-06 18:01:56 -05:00
333e5a9446
Make purge_old_snapshot_archives aware of maximum_incremental_snapshot_archives_to_retain ( #19615 )
2021-09-04 07:37:29 -05:00
89a31ff473
change untar to use unpack instead of unpack_in ( #19216 )
...
* change untar to use unpack instead of unpack_in
* hacky, but maybe passes tests
* chore: bump tar from 0.4.35 to 0.4.37
Bumps [tar](https://github.com/alexcrichton/tar-rs ) from 0.4.35 to 0.4.37.
- [Release notes](https://github.com/alexcrichton/tar-rs/releases )
- [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.35...0.4.37 )
---
updated-dependencies:
- dependency-name: tar
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* cleanup
* cleanup, add validate_inside_dst
* collapse use
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* delete comment line
* add comments
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-08-18 15:49:02 -05:00
d1debcd971
Add incremental snapshot utils ( #18504 )
...
This commit adds high-level functions for creating and loading-from
incremental snapshots, plus all low-level functions required to perform
those tasks. This commit **does not** add taking incremental snapshots
as part of a running validator, nor starting up a node with an
incremental snapshot; just laying ground work.
Additionally, `snapshot_utils` and `serde_snapshot` have been
refactored to use a common code paths for the different snapshots.
Also of note, some renaming has happened:
1. Snapshots are now either `full_` or `incremental_` throughout the
codebase. If not specified, the code applies to both.
2. Bank snapshots now are called "bank snapshots"
(before they were called "slot snapshots", "bank snapshots", or
just "snapshots"). The one exception is within `Bank`, where they
are still just "snapshots", because they are already "bank
snapshots".
3. Snapshot archives now have `_archive` in the code. This
should clear up an ambiguity between bank snapshots and snapshot
archives.
2021-07-22 14:40:37 -05:00
e36247d187
As a last resort try to download an uncompressed snapshot
...
`solana-test-validator` creates uncompressed snapshots and it can be
useful to attach another validator to a `solana-test-validator` cluster
from time to time
2021-07-02 14:35:06 -07:00
0426c2d28e
chore: bump bzip2 from 0.3.3 to 0.4.3 ( #18361 )
...
* chore: bump bzip2 from 0.3.3 to 0.4.3
Bumps [bzip2](https://github.com/alexcrichton/bzip2-rs ) from 0.3.3 to 0.4.3.
- [Release notes](https://github.com/alexcrichton/bzip2-rs/releases )
- [Commits](https://github.com/alexcrichton/bzip2-rs/commits )
---
updated-dependencies:
- dependency-name: bzip2
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Update to new api
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-07-01 13:10:52 -06:00
45d54b1fc6
Add SnapshotArchiveInfo and refactor functions in snapshot_utils ( #18232 )
2021-07-01 12:20:56 -05:00
05924423c2
chore: bump reqwest from 0.11.2 to 0.11.4 ( #18362 )
...
* chore: bump reqwest from 0.11.2 to 0.11.4
Bumps [reqwest](https://github.com/seanmonstar/reqwest ) from 0.11.2 to 0.11.4.
- [Release notes](https://github.com/seanmonstar/reqwest/releases )
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.2...v0.11.4 )
---
updated-dependencies:
- dependency-name: reqwest
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-07-01 09:59:30 -06:00
89a3e4f91e
Move SnapshotConfig into its own module ( #18331 )
...
Also move ArchiveFormat to snapshot_utils, and do not
reexport SnapshotVersion.
2021-07-01 08:55:26 -05:00
78968d132f
chore: bump log from 0.4.11 to 0.4.14 ( #18323 )
...
* chore: bump log from 0.4.11 to 0.4.14
Bumps [log](https://github.com/rust-lang/log ) from 0.4.11 to 0.4.14.
- [Release notes](https://github.com/rust-lang/log/releases )
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/log/compare/0.4.11...0.4.14 )
---
updated-dependencies:
- dependency-name: log
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* Make version consistent
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-06-30 18:32:01 +00:00
8e1e57a181
chore: bump tar from 0.4.28 to 0.4.35 ( #17666 )
...
* chore: bump tar from 0.4.28 to 0.4.35
Bumps [tar](https://github.com/alexcrichton/tar-rs ) from 0.4.28 to 0.4.35.
- [Release notes](https://github.com/alexcrichton/tar-rs/releases )
- [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.28...0.4.35 )
---
updated-dependencies:
- dependency-name: tar
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-06-02 14:17:58 -06:00
a43c29e858
Bump indicatif from 0.15.0 to 0.16.2 ( #17628 )
...
* Bump indicatif from 0.15.0 to 0.16.2
Bumps [indicatif](https://github.com/mitsuhiko/indicatif ) from 0.15.0 to 0.16.2.
- [Release notes](https://github.com/mitsuhiko/indicatif/releases )
- [Commits](https://github.com/mitsuhiko/indicatif/compare/0.15.0...0.16.2 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Fix message types
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-06-01 17:38:02 +00:00
ec1a307a7c
Bump console from 0.11.3 to 0.14.1 ( #16301 )
...
* Bump console from 0.11.3 to 0.14.1
Bumps [console](https://github.com/mitsuhiko/console ) from 0.11.3 to 0.14.1.
- [Release notes](https://github.com/mitsuhiko/console/releases )
- [Changelog](https://github.com/mitsuhiko/console/blob/master/CHANGELOG.md )
- [Commits](https://github.com/mitsuhiko/console/compare/0.11.3...0.14.1 )
Signed-off-by: dependabot[bot] <support@github.com>
* Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-05-27 18:55:27 +00:00
8eab0e8602
Bump version to v1.8.0 ( #17541 )
2021-05-27 08:51:53 -07:00
54f0fc9f0f
Use type alias for DownloadProgress callback ( #17518 )
...
Convert to use type alias for the callback and cascade the changes to callers. Thanks @jeffwashington for the help making it possible.
Changed the closure for the progress update in the validator main to FnMut and modify the abort count in the closure which is more reliable.
2021-05-26 13:26:07 -07:00
4c17243157
snapshot download enhancement ( #17415 )
...
1. Allow the validator bootstrap code to specify the minimal snapshot download speed. If the snapshot download speed is detected below that, a different RPC can be retried. The default is 10MB/sec.
2. To prevent spinning on a number of sub-optimal choices and not making progress, the abort/retry logic is implemented with the following safe guards:
2.1 at maximum we do this retry for 5 times -- this number is configurable with default 5.
2.2 if the download in one notification round (5 second) is more than 2%, do not do retry -- it is not as bad anyway.
2.3 if the remaining estimate time is less than 1 minutes, do not abort retry as it will be done quickly anyway.
2.4 We do this abort/retry logic only at the first notification to avoid wasting download efforts -- the reasoning is being opportunistic and too greedy may not achieve overall shorter download time.
3. The download_snapshot and download_file is modified with the option allowing caller to notified of download progress via a callback. This allows the business logic of retrying to the place it belongs.
2021-05-25 09:32:12 -07:00
a8dca3976b
Refactor genesis download/load/check functions ( #17276 )
...
* Refactor genesis ingest functions
* Consolidate genesis.bin/genesis.tar.bz2 references
2021-05-24 16:45:36 +02:00
9c42a89a43
Issue #17008 -- make snapshot archives to hold on to configurable. ( #17158 )
...
* purge_old_snapshot_archives is changed to take an extra argument 'maximum_snapshots_to_retain' to control the max number of latest snapshot archives to retain. Note the oldest snapshot is always retained as before and is not subjected to this new options.
* The validator and ledger-tool executables are modified with a CLI argument --maximum-snapshots-to-retain. And the options are propagated down the call chains. Their corresponding shell scripts were changed accordingly.
* SnapshotConfig is modified to have an extra field for the maximum_snapshots_to_retain
* Unit tests are developed to cover purge_old_snapshot_archives
2021-05-12 10:32:27 -07:00
e61b4b7d70
dl-utils: use `wide_msg` everywhere for truncation on narrow terminals
2021-04-14 23:50:42 +00:00
02b81dd05d
Update to reqwest 0.11.2
2021-03-18 11:59:41 -07:00
0209d334bd
Add option for separate snapshot location
...
(cherry picked from commit 6126878f509c69e23480a5ec22b3271e2b16e072)
2021-03-17 10:31:01 -07:00
58b980f9cd
Download snapshot files with a `tmp-` prefix so they'll automatically be cleaned up if interrupted
2021-03-17 09:10:10 -07:00
a2eb655322
=1.7.0
2021-03-16 07:51:07 +00:00
0c9ca5522c
Bump version to v1.7.0
2021-03-13 09:01:21 +00:00
24d18b3cf2
docs: add docs links for crates published to crates.io
2021-03-11 08:38:18 +00:00
7f7370c306
Re-allow clippy::integer_arithmetic at crate-level
2021-02-17 13:55:08 -07:00
600ff0d915
calculate hash from store instead of index ( #15034 )
...
* calculate hash from store instead of index
* restore update hash in abs
2021-02-04 09:00:33 -06:00
bf1943e489
Add solana-test-validator --warp-slot argument
2021-01-22 21:17:02 -08:00
7be6770808
Rename CompressionType to ArchiveFormat
2021-01-09 09:07:49 -08:00
90deadb396
Avoid creating "..tmp" files
2020-12-28 08:58:09 -08:00
7b49c85aa7
Fix download speed ( #14291 )
2020-12-27 16:25:41 -08:00
93ae177503
Do not delete ALL other snapshots before downloading a new snapshot
2020-12-21 08:58:52 +00:00
1e977ac025
Bump version to v1.6.0
2020-12-15 18:28:04 +00:00
88d950e5a8
chore: bump log from 0.4.8 to 0.4.11 ( #13691 )
...
* chore: bump log from 0.4.8 to 0.4.11
Bumps [log](https://github.com/rust-lang/log ) from 0.4.8 to 0.4.11.
- [Release notes](https://github.com/rust-lang/log/releases )
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/log/compare/0.4.8...0.4.11 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-12-13 13:48:23 +09:00
7ee0c9e68a
Add non-progress bar download status ( #13370 )
2020-11-04 10:32:27 -08:00
8a119c1483
Bump reqwest/rayon to get past `cargo audit`
2020-10-11 18:51:09 +00:00
11df2e2236
Bump version to v1.5.0
2020-10-08 04:51:36 +00:00
e12ab9d0dd
Bump version to 1.4.0
2020-08-05 12:04:15 -06:00
6ee222363e
Move BankForks to solana_runtime ( #10637 )
...
* Move BankForks to solana_runtime
* Update imports
2020-06-17 15:27:03 +00:00
943ae9793d
Bump indicatif from 0.14.0 to 0.15.0
...
Bumps [indicatif](https://github.com/mitsuhiko/indicatif ) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/mitsuhiko/indicatif/releases )
- [Commits](https://github.com/mitsuhiko/indicatif/compare/0.14.0...0.15.0 )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-06-14 23:58:57 -07:00
48febf153b
Change documentation introduction to indicate Foundation IP transfer ( #10440 )
2020-06-13 14:41:05 -07:00
6cf30c38a4
Bump reqwest from 0.10.5 to 0.10.6
...
Bumps [reqwest](https://github.com/seanmonstar/reqwest ) from 0.10.5 to 0.10.6.
- [Release notes](https://github.com/seanmonstar/reqwest/releases )
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/reqwest/commits )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-06-03 09:01:36 -07:00
b787254bd6
Bump console from 0.10.3 to 0.11.3 ( #10314 )
...
Bumps [console](https://github.com/mitsuhiko/console ) from 0.10.3 to 0.11.3.
- [Release notes](https://github.com/mitsuhiko/console/releases )
- [Commits](https://github.com/mitsuhiko/console/commits/v0.11.3 )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-05-30 07:47:41 -06:00
9eefb2c929
Bump version to 1.3.0
2020-05-28 15:01:13 -07:00
2d3a906d55
Bump tar from 0.4.27 to 0.4.28 ( #10185 )
...
Bumps [tar](https://github.com/alexcrichton/tar-rs ) from 0.4.27 to 0.4.28.
- [Release notes](https://github.com/alexcrichton/tar-rs/releases )
- [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.27...0.4.28 )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-05-22 15:55:40 -07:00
8da2e1b2f7
Bump tar from 0.4.26 to 0.4.27 ( #10150 )
...
Bumps [tar](https://github.com/alexcrichton/tar-rs ) from 0.4.26 to 0.4.27.
- [Release notes](https://github.com/alexcrichton/tar-rs/releases )
- [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.26...0.4.27 )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-05-20 20:04:22 -07:00
d67ad70443
Revert "Bump console from 0.10.1 to 0.11.2 ( #9763 )"
...
This reverts commit 2de999fb61
2020-04-28 10:43:18 -07:00
2de999fb61
Bump console from 0.10.1 to 0.11.2 ( #9763 )
...
Bumps [console](https://github.com/mitsuhiko/console ) from 0.10.1 to 0.11.2.
- [Release notes](https://github.com/mitsuhiko/console/releases )
- [Commits](https://github.com/mitsuhiko/console/compare/0.10.1...0.11.2 )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-04-28 09:36:07 -07:00
c5a7df9221
Bump console from 0.10.0 to 0.10.1 ( #9726 )
...
Bumps [console](https://github.com/mitsuhiko/console ) from 0.10.0 to 0.10.1.
- [Release notes](https://github.com/mitsuhiko/console/releases )
- [Commits](https://github.com/mitsuhiko/console/compare/0.10.0...0.10.1 )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-04-27 09:16:24 -06:00
47ae57610a
Only build x86_64-unknown-linux-gnu on docs.rs
2020-04-16 19:06:17 -07:00
9493de4443
Add snapshot compression option ( #9276 )
2020-04-03 13:13:49 -07:00
d7fa40087c
Bump version to 1.2.0
2020-03-28 09:44:13 -07:00
f987c18a7e
Strictly validate the contents of snapshot/genesis ( #8959 )
...
automerge
2020-03-25 02:46:41 -07:00
bf60345b7a
Remove all snapshots not matching the desired hash
2020-03-17 22:58:01 -07:00
3a501ad69e
Remove all snapshot not matching the desired hash
2020-03-17 22:18:25 -07:00
215650f6e7
Bump console from 0.9.2 to 0.10.0 ( #8786 )
...
Bumps [console](https://github.com/mitsuhiko/console ) from 0.9.2 to 0.10.0.
- [Release notes](https://github.com/mitsuhiko/console/releases )
- [Commits](https://github.com/mitsuhiko/console/commits )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-11 12:30:15 -06:00
75a84ecdae
Bump reqwest from 0.10.1 to 0.10.4 ( #8787 )
...
Bumps [reqwest](https://github.com/seanmonstar/reqwest ) from 0.10.1 to 0.10.4.
- [Release notes](https://github.com/seanmonstar/reqwest/releases )
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.10.1...v0.10.4 )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-11 09:02:40 -06:00
97986a5241
Move download code to download-utils crate ( #8704 )
2020-03-07 07:08:01 -08:00
Giles Cope
Brooks Prumo
Jeff Washington (jwash)
Michael Vines
dependabot[bot]
Lijun Wang
sakridge
Trent Nelson
DimAn
Greg Fitzgerald
dependabot-preview[bot]
R. M. Shea
Ryo Onodera