Commit Graph

670 Commits

Author SHA1 Message Date
Michael Vines df8dab9d2b Native/builtin programs now receive an InvokeContext 2020-10-29 21:45:24 -07:00
Jack May c458d4b213
move Account to solana-sdk (#13198) 2020-10-28 22:01:07 -07:00
Tyera Eulberg 39686ef098
Use bank timestamp to populate Blockstore::blocktime_cf when correction active (#13158) 2020-10-26 19:23:45 +00:00
Michael Vines 959880db60 Remove unused pubkey::Pubkey imports 2020-10-21 19:08:13 -07:00
Michael Vines 7bc073defe Run `codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand` 2020-10-21 19:08:13 -07:00
Ryo Onodera 0776fa05c7
Add ledger-tool dead-slots and improve purge a lot (#13065)
* Add ledger-tool dead-slots and improve purge a lot

* Reduce batch size...

* Add --dead-slots-only and fixed purge ordering
2020-10-21 17:45:21 +00:00
behzad nouri 75d62ca095
improves threads' utilization in processing gossip packets (#12962)
ClusterInfo::process_packets handles incoming packets in a thread_pool:
https://github.com/solana-labs/solana/blob/87311cce7/core/src/cluster_info.rs#L2118-L2134

However, profiling runtime shows that threads are not well utilized and
a lot of the processing is done sequentially.

This commit redistributes the work done in parallel. Testing on a gce
cluster shows 20%+ improvement in processing gossip packets with much
smaller variations.
2020-10-19 19:03:38 +00:00
Tyera Eulberg 359707c85e
Convert Blockstore Rewards cf to protobuf (#12860)
* Add Blockstore protobuf cf type

* Add Rewards message to proto and make generated pub

* Convert Rewards cf to ProtobufColumn

* Add bench

* Adjust tags

* Move solana proto definitions and conversion methods to new crate
2020-10-15 18:04:10 -06:00
Michael Vines 8a119c1483 Bump reqwest/rayon to get past `cargo audit` 2020-10-11 18:51:09 +00:00
Tyera Eulberg b028c47d2b
Correct Bank timestamp drift every slot (#12737)
* Move timestamp helper to sdk

* Add Bank method for getting timestamp estimate

* Return sysvar info from Bank::clock

* Add feature-gated timestamp correction

* Rename unix_timestamp method to be more descriptive

* Review comments

* Add timestamp metric
2020-10-09 21:53:41 +00:00
Ryo Onodera 1f4bcf70b0
Fix various ledger-tool error due to no builtins (#12759)
* Fix various ledger-tool error due to no builtins

* Add missing file...
2020-10-09 12:19:36 -06:00
Josh 8f5431551e
Store program logs in blockstore / bigtable (TransactionWithStatusMeta) (#12678)
* introduce store program logs in blockstore / bigtable

* fix test, transaction logs created for successful transactions

* fix test for legacy bincode implementation around log_messages

* only api nodes should record logs

* truncate transaction logs to 100KB

* refactor log truncate for improved coverage
2020-10-08 12:06:15 -07:00
Michael Vines 11df2e2236 Bump version to v1.5.0 2020-10-08 04:51:36 +00:00
sakridge 71a308affd
Use datapoint instead of print for is_full signal (#12673) 2020-10-07 13:36:54 -07:00
sakridge ce98088457
Track inserted repair shreds (#12455) 2020-09-29 14:13:21 -07:00
Michael Vines 31696a1d72 Port BPFLoader2 activation to FeatureSet and rework built-in program activation 2020-09-28 12:50:19 -07:00
dependabot[bot] 0718501c72 chore: bump rocksdb from 0.14.0 to 0.15.0
Bumps [rocksdb](https://github.com/rust-rocksdb/rust-rocksdb) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/rust-rocksdb/rust-rocksdb/releases)
- [Changelog](https://github.com/rust-rocksdb/rust-rocksdb/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-rocksdb/rust-rocksdb/compare/v0.14.0...v0.15.0)

Signed-off-by: dependabot[bot] <support@github.com>
2020-09-28 09:52:31 -07:00
Michael Vines c10da16d7b Port instructions sysvar and secp256k1 program activation to FeatureSet 2020-09-25 11:40:36 -07:00
Justin Starry 6601ec8f26
Record and store invoked instructions in transaction meta (#12311)
* Record invoked instructions and store in transaction meta

* Enable cpi recording if transaction sender is some

* Rename invoked to innerInstructions
2020-09-24 22:36:22 +08:00
Justin Starry 731a943239
Remove transaction encoding from storage layer (#12404) 2020-09-24 13:10:29 +08:00
sakridge 68e5a2ef56
Add RPC notify and banking keys debug (#12396) 2020-09-23 18:46:42 -07:00
Josh 65a6bfad09
Add blockstore column to store performance sampling data (#12251)
* Add blockstore column to store performance sampling data

* introduce timer and write performance metrics to blockstore

* introduce getRecentPerformanceSamples rpc

* only run on rpc nodes enabled with transaction history

* add unit tests for get_recent_performance_samples

* remove RpcResponse from rpc call

* refactor to use Instant::now and elapsed for timer

* switch to root bank and ensure not negative subraction

* Add PerfSamples to purge/compaction

* refactor to use Instant::now and elapsed for timer

* switch to root bank and ensure not negative subraction

* remove duplicate constants

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-09-22 12:26:32 -07:00
Ryo Onodera cb8661bd49
Persistent tower (#10718)
* Save/restore Tower

* Avoid unwrap()

* Rebase cleanups

* Forcibly pass test

* Correct reconcilation of votes after validator resume

* d b g

* Add more tests

* fsync and fix test

* Add test

* Fix fmt

* Debug

* Fix tests...

* save

* Clarify error message and code cleaning around it

* Move most of code out of tower save hot codepath

* Proper comment for the lack of fsync on tower

* Clean up

* Clean up

* Simpler type alias

* Manage tower-restored ancestor slots without banks

* Add comment

* Extract long code blocks...

* Add comment

* Simplify returned tuple...

* Tweak too aggresive log

* Fix typo...

* Add test

* Update comment

* Improve test to require non-empty stray restored slots

* Measure tower save and dump all tower contents

* Log adjust and add threshold related assertions

* cleanup adjust

* Properly lower stray restored slots priority...

* Rust fmt

* Fix test....

* Clarify comments a bit and add TowerError::TooNew

* Further clean-up arround TowerError

* Truly create ancestors by excluding last vote slot

* Add comment for stray_restored_slots

* Add comment for stray_restored_slots

* Use BTreeSet

* Consider root_slot into post-replay adjustment

* Tweak logging

* Add test for stray_restored_ancestors

* Reorder some code

* Better names for unit tests

* Add frozen_abi to SavedTower

* Fold long lines

* Tweak stray ancestors and too old slot history

* Re-adjust error conditon of too old slot history

* Test normal ancestors is checked before stray ones

* Fix conflict, update tests, adjust behavior a bit

* Fix test

* Address review comments

* Last touch!

* Immediately after creating cleaning pr

* Revert stray slots

* Revert comment...

* Report error as metrics

* Revert not to panic! and ignore unfixable test...

* Normalize lockouts.root_slot more strictly

* Add comments for panic! and more assertions

* Proper initialize root without vote account

* Clarify code and comments based on review feedback

* Fix rebase

* Further simplify based on assured tower root

* Reorder code for more readability

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-19 14:03:54 +09:00
carllin 3533e11786
Fix blockstore processor squash (#12319) 2020-09-17 21:33:08 -07:00
sakridge 3930cb865a
Add keccak-secp256k1 instruction (#11839)
* Implement keccak-secp256k1 instruction

Verifies eth addreses with ecrecover function

* Move secp256k1 test
2020-09-15 18:23:21 -07:00
Ryo Onodera de4a613610
Check bank capitalization (#11927)
* Check bank capitalization

* Simplify and unify capitalization calculation

* Improve and add tests

* Avoid overflow and inhibit automatic restart

* Fix test

* Tweak checked sum for cap. and add tests

* Fix broken build after merge conflicts..

* Rename to ClusterType

* Rename confusing method

* Clarify comment

* Verify cap. in rent and inflation tests

Co-authored-by: Stephen Akridge <sakridge@gmail.com>
2020-09-12 01:48:06 +09:00
Tyera Eulberg 05db41fe9c
Cache block time in Blockstore (#11955)
* Add blockstore column to cache block times

* Add method to cache block time

* Add service to cache block time

* Update rpc getBlockTime to use new method, and refactor blockstore slightly

* Return block_time with confirmed block, if available

* Add measure and warning to cache-block-time
2020-09-09 09:33:14 -06:00
Ryo Onodera c274e26eb8
Prevent unbound memory growth by blockstore_processor (#12110)
* Prevent unbound memory growth by blockstore_processor

* Promote log to info! considering infrequency

* Exclude the time of freeing from interval...

* Skip not-shrinkable slots even if forced

* Add comment
2020-09-09 12:05:10 +09:00
Ryo Onodera 53b8ea4464
Rename to ClusterType and restore devnet compat. (#12068)
* Rename to ClusterType and restore devnet compat.

* De-duplicate parse code and add comments

* Adjust default Devnet genesis & reduce it in tests
2020-09-08 23:55:09 +09:00
Michael Vines cd3c134b58 Remove dead code 2020-09-04 16:01:49 -07:00
Michael Vines d3611f74c8 Add exit flag for bigtable upload operations 2020-09-04 16:01:49 -07:00
Michael Vines bafdcf24f5 Add BigTableUploadService 2020-09-04 16:01:49 -07:00
Michael Vines bc7731b969 Add BigTableUploadService 2020-09-04 16:01:49 -07:00
Michael Vines 91a56caed2 Relocate BigTable uploader to ledger/ crate 2020-09-04 16:01:49 -07:00
carllin 1c1a3f979d
Detect and notify when deserializable shreds are available (#11816)
* Add logic to check for complete data ranges

* Add RPC signature notification

Co-authored-by: Carl <carl@solana.com>
2020-09-01 22:06:06 -07:00
sakridge c4253dc0f9
Update to rayon 1.4.0 (#11898) 2020-08-28 21:00:11 -07:00
Jack May db4bbb3569
Switch programs activation to whole-set based gating (#11750)
* Implement Debug for MessageProcessor

* Switch from delta-based gating to whole-set gating

* Remove dbg!

* Fix clippy

* Clippy

* Add test

* add loader to stable operating mode at proper epoch

* refresh_programs_and_inflation after ancestor setup

* Callback via snapshot; avoid account re-add; Debug

* Fix test

* Fix test and fix the past history

* Make callback management stricter and cleaner

* Fix test

* Test overwrite and frozen for native programs

* Test epoch callback with genesis-programs

* Add assertions for parent bank

* Add tests and some minor cleaning

* Remove unsteady assertion...

* Fix test...

* Fix DOS

* Skip ensuring account by dual (whole/delta) gating

* Fix frozen abi implementation...

* Move compute budget constatnt init back into bank

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-08-26 01:49:15 +09:00
Michael Vines 247f27af37
Submit a vote timestamp every vote (#10630)
* Submit a timestamp for every vote

* Submit at most one vote timestamp per second

* Submit a timestamp for every new vote

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-08-21 11:10:51 -06:00
carllin f7adb68599
Squash supermajority root on blockstore replay at startup (#11727) 2020-08-20 21:56:25 -07:00
Trent Nelson 8841c3398c Bump MacOS nofile recommendation message 2020-08-20 23:57:07 +00:00
carllin b7ed06b17a
Cleanup test utilities (#11723)
* Add voting utility

* Add blockstore utility

Co-authored-by: Carl <carl@solana.com>
2020-08-20 05:04:38 +00:00
Trent Nelson 11951eb009 Sync FD limit and max maps to 500k 2020-08-19 16:36:03 +00:00
Michael Vines d15173ad9d Address latest nightly clippy lints, but globally disable stable_sort_primitive 2020-08-17 22:36:10 -07:00
Michael Vines adc984a225 Rename Binary64 to Base64. Establish Base58 encoding 2020-08-17 17:26:29 -07:00
Michael Vines b5f3ced860 Add base64 (binary64) encoding for getConfirmedTransaction/getConfirmedBlock 2020-08-17 17:26:29 -07:00
Tyera Eulberg 6c5b8f324a
Rpc: Add until parameter for getConfirmedSignaturesForAddress2 (#11644)
* Refactor bigtable apis to accept start and end keys

* Make helper fn to deserialize cell data

* Refactor get_confirmed_signatures_for_address to use get_row_data range

* Add until param to get_confirmed_signatures_for_address

* Add until param to blockstore api

* Plumb until through client/cli

* Simplify client params
2020-08-15 10:42:17 -06:00
Jack May 7c736f71fe
Make BPF Loader static (#11516) 2020-08-14 12:32:45 -07:00
Tyera Eulberg 99fb36fe45
RPC: getConfirmedSignaturesForAddress2 only returns confirmed signatures (#11615)
* Add failing test case

* Limit to only rooted slots
2020-08-13 11:07:42 -06:00
Tyera Eulberg de5fb3ba0e
Blockstore address signatures: handle slots that cross primary indexes, and refactor get_confirmed_signatures_for_address2 (#11497)
* Freeze address-signature index in the middle of slot to show failure case

* Secondary filter on signature

* Use AddressSignatures iterator instead of manually decrementing slots

* Remove unused method

* Add metrics

* Add transaction-status-index doccumentation
2020-08-10 10:27:38 -06:00
carllin 7e25130529
Send votes from banking stage to vote listener (#11434)
*  Send votes from banking stage to vote listener

Co-authored-by: Carl <carl@solana.com>
2020-08-07 11:21:35 -07:00
Tyera Eulberg 5530ee4c95
Add address-based lower bound to get_confirmed_signatures_for_address2 loop (#11426) 2020-08-06 22:29:25 +00:00
Tyera Eulberg 1061b50665
Fix blockstore empty panic (#11423)
* Add panicking test

* Add failing test: fresh transaction-status column shouldn't point at valid root 0

* Prevent transaction status match outside of primary-index bounds

* Initialize transaction-status and address-signature primer entries with Slot::MAX

* Revert "Add failing test: fresh transaction-status column shouldn't point at valid root 0"

This reverts commit cbad2a9fae22e5531e3b4ff1b0a9d6a223826c71.

* Revert "Initialize transaction-status and address-signature primer entries with Slot::MAX"

This reverts commit ffaeac0669d0cbe18dd68b5ce177e15a92360b72.
2020-08-06 22:21:46 +00:00
Greg Fitzgerald 3fdd8ffdf0
Remove circular dep between InstructionError and SystemError (#11427) 2020-08-06 22:04:43 +00:00
Michael Vines 02c0981ecf Rename startAfter to before 2020-08-05 14:33:14 -07:00
Michael Vines a11f137810 Rework get_confirmed_signatures_for_address2 2020-08-05 14:33:14 -07:00
Michael Vines 1b2276520b Add getConfirmedSignaturesForAddress2 RPC method 2020-08-05 14:33:14 -07:00
Michael Vines 494968be66 UiTransactionEncoding is now copy 2020-08-05 14:27:12 -07:00
Trent Nelson e12ab9d0dd Bump version to 1.4.0 2020-08-05 12:04:15 -06:00
Michael Vines eefcf484cb clippy 2020-08-03 18:35:15 +00:00
carllin bf18524368
Add hook for getting vote transactions on replay (#11264)
* Add hook for getting vote transactions on replay

Co-authored-by: Carl <carl@solana.com>
2020-07-29 23:17:40 -07:00
Greg Fitzgerald 8ccce1e543
Upgrade ed25519-dalek (#11183) 2020-07-23 17:23:51 -06:00
Tyera Eulberg 6c38369042
Use OrderedIterator in TransactionStatusService (#11149)
* Split out get-first-err for unit testing

* Add failing test

* Add missing ordering
2020-07-21 20:06:49 +00:00
Tyera Eulberg 9a80e31bae
Expose tss to the other blockstore_processor path (#11070) 2020-07-14 21:14:48 -06:00
dependabot-preview[bot] cee9177ec7 Bump libc from 0.2.71 to 0.2.72
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.71 to 0.2.72.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.71...0.2.72)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-07-13 15:00:27 -07:00
Michael Vines 491f5ae61a Add block time placeholder to getConfirmedBlock 2020-07-10 14:47:11 +00:00
dependabot-preview[bot] 841ecfd927
chore(deps): bump bincode from 1.2.1 to 1.3.1 (#10867)
* chore(deps): bump bincode from 1.2.1 to 1.3.1

Bumps [bincode](https://github.com/servo/bincode) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/servo/bincode/releases)
- [Commits](https://github.com/servo/bincode/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

* Switch from deprecated method

* Add options to maintain behavior with bincode::options()

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-07-09 00:08:05 +00:00
carllin 3f6042d8b3
Add RepairWeight to track votes seen in gossip for weighted repair (#10903)
* Add RepairWeight

Co-authored-by: Carl <carl@solana.com>
2020-07-06 22:49:40 -07:00
sakridge 58a475b789
Add db recovery methods (#10838) 2020-07-06 12:43:45 -07:00
sakridge 658de5b347
Skip and warn for hard-forks which are less than the start slot (#10918)
* Skip and warn for hard-forks which are less than the start slot

Option is used during a restart, but then after the restart is
complete, then the option is not needed if the starting slot
is past the hard-fork since the hard-fork should already be
in the snapshot it started from.

* Update ledger/src/blockstore_processor.rs

Co-authored-by: Michael Vines <mvines@gmail.com>

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-07-05 13:51:43 -07:00
carllin f014c94369
Add breakdown of erasure blobs (#10912)
Co-authored-by: Carl <carl@solana.com>
2020-07-03 17:44:32 -07:00
carllin f17ac70bb2
Add weighted traversal (#10877)
Co-authored-by: Carl <carl@solana.com>
2020-07-02 14:33:04 -07:00
Tyera Eulberg 2669ccb864
Update struct prefixes to Ui (#10874)
* Update comments

* Use Ui prefix
2020-07-01 20:06:40 +00:00
dependabot-preview[bot] 4deec51703
chore(deps): bump rayon from 1.3.0 to 1.3.1 (#10605)
* chore(deps): bump rayon from 1.3.0 to 1.3.1

Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.3.0 to 1.3.1.
- [Release notes](https://github.com/rayon-rs/rayon/releases)
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md)
- [Commits](https://github.com/rayon-rs/rayon/compare/rayon-core-v1.3.0...v1.3.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-07-01 08:09:28 -07:00
Ryo Onodera b89e506cbb
Consistently create temp dirs under ledger/farf (#10848) 2020-07-01 11:38:59 +09:00
sakridge 583cec922b
Remove ledger purge batching (#10830) 2020-06-29 14:44:35 -07:00
carllin 0fde0d7379
Add ancestor iterator to lib.rs (#10813)
Co-authored-by: Carl <carl@solana.com>
2020-06-25 23:39:55 +00:00
Greg Fitzgerald 1c498369b5
Remove fee-payer guesswork from Message and Transaction (#10776)
* Make Message::new_with_payer the default constructor

* Remove Transaction::new_[un]signed_instructions

These guess the fee-payer instead of stating it explicitly
2020-06-24 14:52:38 -06:00
sakridge 0e393a5684
Allow for hard fork at last root (#10762) 2020-06-23 21:49:11 -07:00
sakridge a1ef921b88
Rework backup and clear function (#10751) 2020-06-23 21:29:07 +00:00
carllin 77b8de193c
Add utility functions for testing (#10749)
* Add ancestor iterator

* Add blockstore generation from trees

Co-authored-by: Carl <carl@solana.com>
2020-06-23 12:05:00 -07:00
sakridge 2ba8fc5243
Remove slots past wait-for-supermajority slot. (#10720) 2020-06-22 20:27:25 -07:00
carllin a33fef9af2
Add repair breakdown by slot and index (#10717)
* Slot full logging

* Repair stats logging

Co-authored-by: Carl <carl@solana.com>
2020-06-19 18:28:15 -07:00
Greg Fitzgerald 0550b893b0
Fix typos (#10675)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-06-17 20:54:52 -07:00
Greg Fitzgerald 6ee222363e
Move BankForks to solana_runtime (#10637)
* Move BankForks to solana_runtime

* Update imports
2020-06-17 15:27:03 +00:00
Kristofer Peterson 50c93d4441
Cleanup error type mapping in serde_snapshot (#10580)
* Cleanup error type mapping in serde_snapshot

* Fail when account storage entry files cannot be moved from snapshot
2020-06-17 17:56:29 +09:00
sakridge 0de6c444d6
Simd poh (#10604)
* Simd poh

* Fix poh verify bench
2020-06-16 23:03:26 -07:00
sakridge 1eca9b19ab
Entry verify cleanup and gossip counters (#10632)
* Add prune message counter

* Switch to us verification time to match other counters

* Add separate transaction/poh verify timing
2020-06-16 14:00:29 -07:00
dependabot-preview[bot] 480f69cc5e Bump serde from 1.0.111 to 1.0.112
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.111 to 1.0.112.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.111...v1.0.112)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-06-15 07:58:16 -07:00
R. M. Shea 48febf153b
Change documentation introduction to indicate Foundation IP transfer (#10440) 2020-06-13 14:41:05 -07:00
Greg Fitzgerald 2eb6f498a8
Remove redundant BankForks parameter (#10537) 2020-06-12 11:04:17 -06:00
Kristofer Peterson e23340d89e
Clippy cleanup for all targets and nighly rust (also support 1.44.0) (#10445)
* address warnings from 'rustup run beta cargo clippy --workspace'

minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs

expect some performance improvement AccountsDB::clean_accounts()

* address warnings from 'rustup run beta cargo clippy --workspace --tests'

* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'

* rustfmt

* fix warning stragglers

* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned

* Remove lock around JsonRpcRequestProcessor (#10417)

automerge

* make ancestors parameter optional to avoid forcing construction of empty hash maps

Co-authored-by: Greg Fitzgerald <greg@solana.com>
2020-06-09 09:38:14 +09:00
sakridge 2725acebea
Lower counter level (#10428) 2020-06-05 15:53:09 -07:00
Michael Vines aa6832964c
ledger_cleanup_service: compact at a slower rate than purging (#10414) 2020-06-04 21:06:06 -07:00
dependabot-preview[bot] f901571026 Bump serde from 1.0.110 to 1.0.111
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.110 to 1.0.111.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.110...v1.0.111)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-06-02 23:29:14 -07:00
Ryo Onodera caa7f7a0c9
Support opening an in-use rocksdb as secondary (#10209)
automerge
2020-06-02 21:32:44 -07:00
Tyera Eulberg eee9a08376
Purge TransactionStatus and AddressSignatures exactly from ledger-tool (#10358)
* Add failing test

* Add execution path to purge primary-index columns exactly

* Fail gracefully if older TransactionStatus rocksdb keys are present

* Remove columns_empty check for special columns

* Move blockstore purge methods to submodule

* Remove unused column empty check
2020-06-02 19:49:31 -06:00
Greg Fitzgerald 19d11800bf
Remove WithSigner (#10325)
automerge
2020-05-29 23:17:44 -07:00
carllin bc37dfc4c1
log leader (#10316)
automerge
2020-05-29 04:35:20 -07:00
Kristofer Peterson fb4d8e1f62
cleanup clippy tests (#10172)
automerge
2020-05-29 00:26:06 -07:00
dependabot-preview[bot] a71833c17b
Bump regex from 1.3.8 to 1.3.9 (#10308)
* Bump regex from 1.3.8 to 1.3.9

Bumps [regex](https://github.com/rust-lang/regex) from 1.3.8 to 1.3.9.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.3.8...1.3.9)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-05-28 21:48:39 -07:00
Michael Vines 9eefb2c929 Bump version to 1.3.0 2020-05-28 15:01:13 -07:00
dependabot-preview[bot] 9227874ada
Bump regex from 1.3.7 to 1.3.8 (#10293)
* Bump regex from 1.3.7 to 1.3.8

Bumps [regex](https://github.com/rust-lang/regex) from 1.3.7 to 1.3.8.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/regex-1.3.7...1.3.8)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-05-28 10:02:23 -07:00
Michael Vines 5ac2ae1178
Purge next slots to avoid a blockstore_processor panic on restart (#10281) 2020-05-27 19:55:40 -07:00
Michael Vines e3b444834f
Add more logging while unpacking snapshots (#10266) 2020-05-27 10:41:05 -07:00
Michael Vines 7abdfc23f9
succceed is not a word 2020-05-26 23:17:50 -07:00
dependabot-preview[bot] 1bfc4c1489
Bump libc from 0.2.70 to 0.2.71 (#10249)
* Bump libc from 0.2.70 to 0.2.71

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.70 to 0.2.71.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.70...0.2.71)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2020-05-26 11:04:15 -07:00
Michael Vines 156387aba4
LedgerCleanupService no longer causes an OOM and actually purges (#10199)
* cleanup_ledger() now services new_root_receiver while purging
* purge_slots() now fully deletes before compacting
* Add ledger pruning grafana graph
2020-05-24 21:41:54 -07:00
dependabot-preview[bot] 8a8384e674
Bump sha2 from 0.8.1 to 0.8.2 (#10205)
* Bump sha2 from 0.8.1 to 0.8.2

Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.8.1 to 0.8.2.
- [Release notes](https://github.com/RustCrypto/hashes/releases)
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.8.1...sha2-v0.8.2)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-05-24 19:44:04 -07:00
dependabot-preview[bot] 2d3a906d55
Bump tar from 0.4.27 to 0.4.28 (#10185)
Bumps [tar](https://github.com/alexcrichton/tar-rs) from 0.4.27 to 0.4.28.
- [Release notes](https://github.com/alexcrichton/tar-rs/releases)
- [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.27...0.4.28)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-05-22 15:55:40 -07:00
Kristofer Peterson b7a32f01c0
Multi-version snapshot support (#9980)
* Multi-version snapshot support

* rustfmt

* Remove CLI options and runtime support for selection output snapshot version.
Address some clippy complaints.

* Muzzle clippy type complexity warning.

Despite clippy's suggestion, it is not currently possible to create type aliases
for traits and so everything within the 'Box<...>' cannot be type aliased.

This then leaves creating full blown traits, and either implementing
said traits by closure (somehow) or moving the closures into new structs
implementing said traits which seems a bit of a palaver.

Alternatively it is possible to define and use the type alias 'type ResultBox<T> = Result<Box<T>>'
which does seems rather pointless and not a great reduction in complexity but is enough to keep clippy happy.

In the end I simply went with squelching the clippy warning.

* Remove now unused Serialize/Deserialize trait implementations for AccountStorageEntry and AppendVec

* refactor versioned de/serialisers

* rename serde_utils to serde_snapshot

* move call to accounts_db.generate_index() back down to context_accountsdb_from_stream()

* update version 1.1.1 to 1.2.0
remove nested use of serialize_bytes

* cleanups

* Add back measurement of account storage entry serialization.
Remove construction of Vec and HashMap temporaries during serialization.

* consolidate serialisation test cases into serde_snapshot.
clean up leakage of implementation details in serde_snapshot.

* move short term / legacy snapshot code into child module

* add serialize_iter_as_tuple

* preliminary integration of following commit

commit 6d58b73c47294bfb93465d5a83cd2175660b6e6d
Author: Ryo Onodera <ryoqun@gmail.com>
Date:   Wed May 20 14:02:02 2020 +0900

    Confine snapshot 1.1 relic to versioned codepath

* refactored serde_snapshot, rustfmt
legacy accounts_db format now "owns" both leading u64s, legacy bank_rc format has none

* reduce type complexity (clippy)
2020-05-23 02:54:24 +09:00
dependabot-preview[bot] 8da2e1b2f7
Bump tar from 0.4.26 to 0.4.27 (#10150)
Bumps [tar](https://github.com/alexcrichton/tar-rs) from 0.4.26 to 0.4.27.
- [Release notes](https://github.com/alexcrichton/tar-rs/releases)
- [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.26...0.4.27)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-05-20 20:04:22 -07:00
carllin 439fd30840
Fix erasure (#10095)
* Fix bad FEC blocks

* Add test

Co-authored-by: Carl <carl@solana.com>
2020-05-19 16:13:12 -07:00
carllin 97f2bcff69
master: Add nonce to shreds repairs, add shred data size to header (#10109)
* Add nonce to shreds/repairs

* Add data shred size to header

Co-authored-by: Carl <carl@solana.com>
2020-05-19 12:38:18 -07:00
sakridge f562ed4cc8 Distinguish between shred type in shred fetch stage duplicate filter (#10068)
* Shred type check

* Test
2020-05-15 13:23:56 -07:00
Greg Fitzgerald 5e89bd8868
Panic if no fee-payer found via Message::new() (#10050)
automerge
2020-05-15 12:23:09 -07:00
Michael Vines 7080fb9b37
Abort if the open fd limit cannot be increased (#10064)
automerge
2020-05-15 12:14:21 -07:00
Kristofer Peterson 58ef02f02b
9951 clippy errors in the test suite (#10030)
automerge
2020-05-15 09:35:43 -07:00
dependabot-preview[bot] 6a6c5f196a
Bump libc from 0.2.69 to 0.2.70 (#10006)
* Bump libc from 0.2.69 to 0.2.70

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.69 to 0.2.70.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.69...0.2.70)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-05-15 06:18:43 -06:00
sakridge 9575afc8fa
Refactor blockstore recovery code (#10008) 2020-05-13 10:09:38 -07:00
Ryo Onodera 1eb40c3fe0
Introduce eager rent collection (#9527)
* Switch AccountsIndex.account_maps from HashMap to BTreeMap

* Introduce eager rent collection

* Start to add tests

* Avoid too short eager rent collection cycles

* Add more tests

* Add more tests...

* Refacotr!!!!!!

* Refactoring follow up

* More tiny cleanups

* Don't rewrite 0-lamport accounts to be deterministic

* Refactor a bit

* Do hard fork, restore tests, and perf. mitigation

* Fix build...

* Refactor and add switch over for testnet (TdS)

* Use to_be_bytes

* cleanup

* More tiny cleanup

* Rebase cleanup

* Set Bank::genesis_hash when resuming from snapshot

* Reorder fns and clean ups

* Better naming and commenting

* Yet more naming clarifications

* Make prefix width strictly uniform for 2-base partition_count

* Fix typo...

* Revert cluster-dependent gate

* kick ci?

* kick ci?

* kick ci?
2020-05-13 16:22:14 +09:00
sakridge 903a8a3196
Add retransmit packets_by_slot metrics (#9975) 2020-05-11 13:49:10 -07:00
dependabot-preview[bot] 288c9751c1
Bump serde from 1.0.106 to 1.0.110 (#9964)
* Bump serde from 1.0.106 to 1.0.110

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.106 to 1.0.110.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.106...v1.0.110)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-05-10 16:24:25 -07:00
carllin ad3c8fb812
More logging around failure (#9967)
automerge
2020-05-10 16:01:20 -07:00
dependabot-preview[bot] c970bbea4f
Bump serde_bytes from 0.11.3 to 0.11.4 (#9914)
Bumps [serde_bytes](https://github.com/serde-rs/bytes) from 0.11.3 to 0.11.4.
- [Release notes](https://github.com/serde-rs/bytes/releases)
- [Commits](https://github.com/serde-rs/bytes/compare/0.11.3...0.11.4)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-05-07 23:26:37 -07:00
Trent Nelson f12c6c1ed1
BSD compat for 9493de4 (#9922)
automerge
2020-05-07 16:38:33 -07:00
Michael Vines 09ae61651a
Eliminate BankForksInfo (#9887) 2020-05-06 08:24:59 -07:00
Michael Vines 72312ad615
Avoid holding the entire rooted path while loading bank forks (#9885) 2020-05-05 19:45:41 -07:00
carllin 3442f36f8a
Repair alternate versions of dead slots (#9805)
Co-authored-by: Carl <carl@solana.com>
2020-05-05 14:07:21 -07:00
Ryo Onodera a91236012d
Pass around --max-genesis-archive-unpacked-size (#9161)
automerge
2020-04-29 18:53:34 -07:00
Stephen Akridge eb24f3df84 Update dalek version 2020-04-28 21:02:47 -06:00
Tyera Eulberg a12428a5b8
Use Blockstore lowest_slot to start root iterator (#9738) 2020-04-28 10:22:10 -06:00
Jack May efad193180
Make default programs static (#9717) 2020-04-27 21:05:12 -07:00
Michael Vines c11abf88b7
Clean up `use` to keep rust 1.43.0 from complaining (#9740) 2020-04-27 16:54:11 -07:00
Tyera Eulberg a7f33b5014
Cache banks in BankForks until optional largest_confirmed_root (#9678)
automerge
2020-04-24 15:49:57 -07:00
sakridge fa20963b93
Revert shred fs (#9712)
* Revert "Untar is called for shred archives that do not exist. (#9565)"

This reverts commit 729cb5eec6.

* Revert "Dont insert shred payload into rocksdb (#9366)"

This reverts commit 5ed39de8c5.
2020-04-24 15:04:23 -07:00
Greg Fitzgerald 76b1c2baf0
One less alloc per transaction (#9705)
* One less alloc per transaction

* Fix benches

* Fix compiler warnings in bench build

* Fix move build

* Fix bench
2020-04-24 13:03:46 -06:00
dependabot-preview[bot] d2cef8ed9b
Bump num_cpus from 1.12.0 to 1.13.0 (#9682)
* Bump num_cpus from 1.12.0 to 1.13.0

Bumps [num_cpus](https://github.com/seanmonstar/num_cpus) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases)
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.12.0...v1.13.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-04-23 13:08:06 -07:00
sakridge 504160b11f
Update to rocksdb 0.14 and set max wal size (#9668) 2020-04-23 08:38:09 -07:00
sakridge ecb343c23b
reduce errors (#9669)
Co-authored-by: Anatoly Yakovenko <anatoly@solana.com>
2020-04-22 19:46:06 -07:00
Tyera Eulberg 5d2f488004
Add getLowestNonpurgedBlock rpc; use blockstore api in getConfirmedBlocks (#9656)
automerge
2020-04-22 13:33:06 -07:00
dependabot-preview[bot] b1ca74ed30 Bump regex from 1.3.6 to 1.3.7
Bumps [regex](https://github.com/rust-lang/regex) from 1.3.6 to 1.3.7.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.3.6...regex-1.3.7)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-04-20 08:02:22 -07:00
sakridge a5419fe79e
Error for invalid shred. (#9588) 2020-04-19 21:15:09 -07:00
anatoly yakovenko 729cb5eec6
Untar is called for shred archives that do not exist. (#9565)
automerge
2020-04-18 08:34:55 -07:00
Michael Vines 47ae57610a Only build x86_64-unknown-linux-gnu on docs.rs 2020-04-16 19:06:17 -07:00
anatoly yakovenko 5ed39de8c5
Dont insert shred payload into rocksdb (#9366)
automerge
2020-04-16 18:20:55 -07:00
sakridge 66abe45ea1
Decouple accounts hash calculation from snapshot hash (#9507) 2020-04-16 15:12:20 -07:00
Stephen Akridge 30b3862770 Don't unwrap on session new 2020-04-16 08:25:45 -07:00
Tyera Eulberg 530c542002
Rpc: Speed up getBlockTime (#9510)
* Add get-block-time metrics

* Add datapoints to blockstore rpc apis

* Tune timestamp_slot_range

* Refactor get_timestamp_slots

* Cargo.lock
2020-04-15 18:09:14 -06:00
Ryo Onodera c77ed82caa
Use same max_age regardless of leader/not-leader (#9423)
automerge
2020-04-14 00:34:41 -07:00
Ryo Onodera 2d0224b64e
Switch reed-solomon-erasure from vendored to upstream (#9483) 2020-04-14 14:33:21 +09:00
dependabot-preview[bot] a0f0e199b7 Bump libc from 0.2.68 to 0.2.69
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.68 to 0.2.69.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.68...0.2.69)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-04-13 17:30:02 -07:00
Tyera Eulberg 91159ea8e3
Rpc: Add getConfirmedSignaturesForAddress (#9407)
automerge
2020-04-09 20:21:31 -07:00
Michael Vines d5a9ee97f2
Add --allow-dead-slots argument to `slot`/`print`/`json` commands (#9408)
automerge
2020-04-09 20:10:51 -07:00
Michael Vines f3afe5c99c
Remove dead code (#9404)
automerge
2020-04-09 13:09:59 -07:00
Tyera Eulberg e1aa247548
Rpc: Add getConfirmedTransaction (#9381)
* Add blockstore method to return a complete transaction by signature

* Plumb getConfirmedTransaction rpc

* Add doc
2020-04-09 00:57:30 -06:00
Tyera Eulberg f655b3f0fd
Fix lowest_cleanup_slot check in Blockstore (#9382)
automerge
2020-04-08 18:47:16 -07:00
carllin 4522e85ac4
Add Metrics/Dashboards tracking block production (#9342)
* Add metric tracking blocks/dropped blocks

Co-authored-by: Carl <carl@solana.com>
2020-04-08 14:35:24 -07:00
Tyera Eulberg 36e73cada4
Add blockstore address-to-signature index (#9367)
automerge
2020-04-08 12:50:39 -07:00
Tyera Eulberg acf64f8476
Update getSignatureStatuses to return historical statuses (#9314)
automerge
2020-04-06 03:04:54 -07:00
Tyera Eulberg 49e2cc6593
Rework TransactionStatus index in blockstore (#9281)
automerge
2020-04-04 20:24:06 -07:00
Michael Vines ad0997e15f
RPC: add `err` field to TransactionStatus, alongside the now deprecated `status` field (#9296)
automerge
2020-04-04 16:13:26 -07:00
dependabot-preview[bot] 8cdf406dd3 Bump serde from 1.0.105 to 1.0.106
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.105 to 1.0.106.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.105...v1.0.106)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-04-04 15:13:52 -07:00
Michael Vines b557b3170e Add log before opening database 2020-04-03 14:55:06 -07:00
sakridge 9493de4443
Add snapshot compression option (#9276) 2020-04-03 13:13:49 -07:00
Jack May d61191db40
fix bench warnings (#9277) 2020-04-02 21:56:38 -07:00
sakridge 73e99cc513
Ledger cleanup fixes (#9131)
* Fix purging happening every slot when cleanup service is not started at slot 0
* Purge by shred count instead of slots since slots can have variable
number of shreds
2020-03-30 19:02:12 -07:00
Michael Vines d7fa40087c
Bump version to 1.2.0 2020-03-28 09:44:13 -07:00
sakridge 2863f8ec65
Use 1gb as genesis limit to fix bench-tps ledger from not starting (#9133)
automerge
2020-03-27 16:50:19 -07:00
Michael Vines b130c298df
Remove chatty 'setting snapshot root:' info log (#9122) 2020-03-27 10:24:59 -07:00
sakridge b7b4aa5d4d
move rpc types from client to client-types crate (#9039)
* Separate client types into own crate, so ledger does not need it

Removes about 50 crates of dependency from ledger

* Drop Rpc name from transaction-status types
2020-03-26 13:29:30 -07:00
sakridge ed036b978d
Accumulate blockstore metrics and submit every 2s (#9075) 2020-03-26 12:51:41 -07:00
Michael Vines b8fda9d730
Log how much data the ledger holds before processing it (#9079) 2020-03-25 21:41:50 -07:00
dependabot-preview[bot] 095c79e863
Bump regex from 1.3.5 to 1.3.6 (#9055)
Bumps [regex](https://github.com/rust-lang/regex) from 1.3.5 to 1.3.6.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.3.5...1.3.6)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-25 17:24:14 -07:00
carllin 9dc69d9843
Store and compute node/stake state in EpochStakes struct (#8958)
* Store and compute needed bank state in EpochStakes struct
2020-03-25 12:19:15 -07:00
Ryo Onodera f987c18a7e
Strictly validate the contents of snapshot/genesis (#8959)
automerge
2020-03-25 02:46:41 -07:00
Michael Vines 70c167182a
ledger tool now outputs transaction status information if available (#9024)
automerge
2020-03-23 12:49:21 -07:00
sakridge d75a470ffa
Ledger processing speed tracking (#9005) 2020-03-23 12:19:11 -07:00
Tyera Eulberg 1b8f9e75dd
Update getSignatureStatus: support multiple signatures, include slot in each response item (#9022)
* Rename enable-rpc-get-confirmed-block

* Rename RpcTransactionStatus -> RpcTransactionStatusMeta

* Return simplified RpcTransactionStatus; Add support for multiple transactions

* Update docs

* typo
2020-03-23 11:25:39 -06:00
sakridge 4d2b83d01f
Add option to disable rocks compaction (#9011) 2020-03-23 08:42:32 -07:00
Michael Vines 88ba8439fc
Add frozen account support (#8989)
automerge
2020-03-22 11:10:04 -07:00
sakridge 4dd0367136
Rwlock storage opt (#9006)
* Remove unecessary account paths rwlock

* Remove path rwlock in accounts_db and optimize storage critical section
2020-03-22 10:04:03 -07:00
carllin dc1db33ec9
Add Capabilities to Signal BroadcastStage to Retransmit (#8899) 2020-03-19 23:35:01 -07:00
Michael Vines f78a90bce2
Vote InitializeAccount and UpdateNode instructions now need a signature from the validator identity (#8947)
automerge
2020-03-19 01:58:52 -07:00
dependabot-preview[bot] 24d871b529
Bump serde from 1.0.104 to 1.0.105 (#8954)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.104 to 1.0.105.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.104...v1.0.105)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-18 22:35:30 -07:00
dependabot-preview[bot] 8f5ee6832f
Bump libc from 0.2.67 to 0.2.68 (#8915)
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.67 to 0.2.68.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.67...0.2.68)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-18 13:27:08 -06:00
Michael Vines bf60345b7a
Remove all snapshots not matching the desired hash 2020-03-17 22:58:01 -07:00
sakridge 7079559c2d
Fix windows build by removing sys-info (#8860)
Doesn't build for windows.
2020-03-16 12:53:13 -07:00
Carl 009c124fac Remove generic 2020-03-16 07:57:07 -07:00
Carl 7029c88305 use matches macro 2020-03-16 07:57:07 -07:00
Carl 9411fc00b8 Lower error level 2020-03-16 07:57:07 -07:00
Michael Vines b7b36bb0a4
Upgrade to Rust 1.42 (#8836)
* Upgrade to Rust 1.42

* deref

* parens

Co-authored-by: Trent Nelson <trent@solana.com>
2020-03-13 14:15:22 -06:00
dependabot-preview[bot] 52b254071c
Bump regex from 1.3.4 to 1.3.5 (#8830)
Bumps [regex](https://github.com/rust-lang/regex) from 1.3.4 to 1.3.5.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.3.4...1.3.5)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-13 07:43:07 -06:00
dependabot-preview[bot] f93c8290f4
Bump sys-info from 0.5.9 to 0.5.10 (#8810)
Bumps [sys-info](https://github.com/FillZpp/sys-info-rs) from 0.5.9 to 0.5.10.
- [Release notes](https://github.com/FillZpp/sys-info-rs/releases)
- [Changelog](https://github.com/FillZpp/sys-info-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/FillZpp/sys-info-rs/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-12 13:42:31 -06:00
Jack May 6eb4973780
Don't use move semantics if not needed (#8793) 2020-03-11 14:37:23 -07:00
dependabot-preview[bot] e41ff2df66
Bump chrono from 0.4.10 to 0.4.11 (#8755)
Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.10 to 0.4.11.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/master/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.10...v0.4.11)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-10 08:43:18 -06:00
dependabot-preview[bot] f88b79d42b
Bump itertools from 0.8.2 to 0.9.0 (#8756)
Bumps [itertools](https://github.com/bluss/rust-itertools) from 0.8.2 to 0.9.0.
- [Release notes](https://github.com/bluss/rust-itertools/releases)
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/bluss/rust-itertools/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-10 08:42:11 -06:00
dependabot-preview[bot] 1a0dd53450
Bump rayon from 1.2.0 to 1.3.0 (#8757)
Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.2.0 to 1.3.0.
- [Release notes](https://github.com/rayon-rs/rayon/releases)
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md)
- [Commits](https://github.com/rayon-rs/rayon/compare/v1.2.0...rayon-core-v1.3.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-10 08:40:02 -06:00
carllin 9872430bd2
Add VoteTracker for tracking cluster's votes in gossip (#8327)
Track votes by slot in cluster_vote_listener
2020-03-09 22:03:09 -07:00
dependabot-preview[bot] ff74452ef3
Bump libc from 0.2.66 to 0.2.67 (#8680)
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.66 to 0.2.67.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.66...0.2.67)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-08 21:38:58 -07:00
Greg Fitzgerald 2242b1b4a5
Bump byteorder from 1.3.2 to 1.3.4 (#8159)
Bumps [byteorder](https://github.com/BurntSushi/byteorder) from 1.3.2 to 1.3.4.
- [Release notes](https://github.com/BurntSushi/byteorder/releases)
- [Changelog](https://github.com/BurntSushi/byteorder/blob/master/CHANGELOG.md)
- [Commits](https://github.com/BurntSushi/byteorder/compare/1.3.2...1.3.4)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-03-05 17:07:18 -07:00
carllin f47a789b15
Add find_incomplete_slots (#8654)
* Add find_incomplete_slots

* Add live slots iterator
2020-03-05 10:58:00 -08:00
Grimes 01607b9860
Add NextSlotsIterator (#8652)
automerge
2020-03-04 20:46:58 -08:00
sakridge 23d8c7ff0e
Generate a snapshot at synchronized points (#8532)
Co-authored-by: anatoly yakovenko <anatoly@solana.com>
2020-03-04 19:23:40 -08:00
carllin dec3da8f9d
Add orphan iterator (#8636) 2020-03-04 18:10:30 -08:00
sakridge b68b74ac32
Check transaction signatures in entry verify (#8596) 2020-03-03 20:49:51 -08:00
sakridge b084c1d437
Remove accounts hack and correctly restore accounts store counts (#8569)
* Remove accounts hack and correctly restore append-vec counts

* Add test
2020-03-03 20:48:55 -08:00
Ryo Onodera d86103383a
Do periodic inbound cleaning for rooted slots (#8436)
* Do periodic inbound compaction for rooted slots

* Add comment

* nits

* Consider not_compacted_roots in cleanup_dead_slot

* Renames in AccountsIndex

* Rename to reflect expansion of removed accounts

* Fix a comment

* rename

* Parallelize clean over AccountsIndex

* Some niceties

* Reduce locks and real chunked parallelism

* Measure each step for sampling opportunities

* Just noticed par iter is maybe lazy

* Replace storage scan with optimized index scan

* Various clean-ups

* Clear uncleared_roots even if no updates
2020-03-03 14:57:25 +09:00
Ryo Onodera 5d9130a3c4 Hack to skip cleanup_dead_slots upon snapshot load 2020-03-02 10:24:12 -07:00
Michael Vines 3acf956f6f Fix test_concurrent_snapshot_packaging 2020-02-26 23:32:53 -07:00
Michael Vines 87b13bef8e Remove bank_slot_from_archive 2020-02-26 23:32:53 -07:00
Michael Vines 0d4cb252c4 Adapt local-cluster/ 2020-02-26 23:32:53 -07:00
Michael Vines fcabc6f799 Rename snapshot.tar.bz2 to snapshot-<slot>-<hash>.tar.bz2 2020-02-26 23:32:53 -07:00
Michael Vines 848c43a9ab Peg snapshot version to 1.0.0 2020-02-26 22:44:39 -07:00
carllin 7a2bf7e7eb
Limit leader schedule search space (#8468)
* Limit leader schedule search space

* Fix and add test

* Rename
2020-02-26 13:35:50 -08:00
Michael Vines fbf78b83c4 Add retry mechanism when downloading genesis and snapshots 2020-02-26 14:21:37 -07:00
Michael Vines 73063544bd Move shred_version module to sdk/ 2020-02-24 14:46:12 -07:00
Michael Vines 90240bf11d r 2020-02-24 14:45:32 -07:00
sakridge b106d3ba60
Fix local cluster test, check for accounts hash (#8411) 2020-02-24 10:23:47 -08:00
sakridge 947a339714
Add snapshot hash of full accounts state (#8295)
* Add snapshot hash of full accounts state

* Use normal hashing for the accounts delta state

* Add merkle
2020-02-22 13:46:40 -08:00
dependabot-preview[bot] 72ae82fe47
Bump crossbeam-channel from 0.3.9 to 0.4.2 (#8400)
Bumps [crossbeam-channel](https://github.com/crossbeam-rs/crossbeam) from 0.3.9 to 0.4.2.
- [Release notes](https://github.com/crossbeam-rs/crossbeam/releases)
- [Changelog](https://github.com/crossbeam-rs/crossbeam/blob/v0.4.2/CHANGELOG.md)
- [Commits](https://github.com/crossbeam-rs/crossbeam/compare/crossbeam-channel-0.3.9...v0.4.2)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-02-22 09:53:38 -07:00
Dan Albert 2d9d2f1e99
Update cargo versions from 1.0 to 1.1 (#8397) 2020-02-21 23:09:45 -08:00
sakridge b7386f9d84
Add --trusted-validator support for snapshot hash validation (#8390) 2020-02-21 18:42:24 -08:00
Michael Vines 48f58a88bc Bump version to 1.0.0 2020-02-20 23:52:19 -07:00
Ryo Onodera d238371b0c
Correct missing entry handling to avoid bad warns (#8339)
* Correct missing entry handling to avoid bad warns

* Pass storage entries to AccountStorageSerialize

* Fix CI.....

* Add tests and reorder condition for cheapest first

* Remove unneeded reference
2020-02-21 15:27:55 +09:00
Michael Vines 18fd52367e
If the node was loaded from a snapshot, advertise it in gossip (#8364)
automerge
2020-02-20 18:53:26 -08:00
Ryo Onodera 5ef06a9d36
Add non-bz2 snapshot for faster creation for dev. (#8350)
* Add non-bz2 snapshot for faster creation for dev.

* Fix tests..

* Revert and always just use snapshot.tar.bz2
2020-02-21 10:19:45 +09:00
Tyera Eulberg ab361a8073
Rename KeypairUtil to Signer (#8360)
automerge
2020-02-20 13:28:55 -08:00
Greg Fitzgerald e8124324ff
Support transaction signing by heterogenous lists of keypairs (#8342)
automerge
2020-02-20 12:13:23 -08:00
sakridge 1720fe6a46
Snapshot hash gossip changes (#8358) 2020-02-20 11:46:13 -08:00
Michael Vines 83a8e82626 Remove dead code 2020-02-18 21:08:43 -07:00
anatoly yakovenko 17fb8258e5
Datapoints overwhelm the metrics queue and blow up ram usage. (#8272)
automerge
2020-02-14 11:11:55 -08:00
Michael Vines ab475e4849 get_confirmed_block: expect() less 2020-02-13 10:56:34 -07:00
Greg Fitzgerald 127553ce4b
Wrap ed25519_dalek::Keypair (#8247) 2020-02-12 14:15:12 -07:00
Michael Vines 72b11081a4 Report validator rewards in getConfirmedBlock JSON RPC 2020-02-11 17:25:45 -07:00
Ryo Onodera 46b6cedff4
Fix nightly clippy warnings (#8199)
automerge
2020-02-10 22:48:50 -08:00
Michael Vines 84b3e12e1f Minor logging improvements 2020-02-10 10:42:42 -07:00
Ryo Onodera 485806c488 Just define BnakSlotDelta type alias (#8186)
automerge
2020-02-10 03:11:37 -08:00
Michael Vines 8b5598fabd
Surface shred version more in tools (#8163)
automerge
2020-02-07 08:57:54 -08:00
Trent Nelson 5b070ad014
CLI: Support offline fee payers (#8009)
* CLI: Support offline fee-payer

* Add some knobs to test genesis/validator helpers

* Add tests
2020-02-07 09:14:26 -07:00
François Garillot f016c9a669
Maintenance : simplify a few patterns, remove unneeded dependencies (#8137)
* Simplify a few pattern matches

* Removing unneeded dependencies, upgrading internal version #s

 net-shaper: Removing log, semver, serde_derive
 bench-tps: Removing serde, serde_derive
 banking-bench: Removing solana
 ledger-tool: Removing bincode, serde, serde_derive
 librapay: Removing solana, language_e2e_tests
 log-analyzer: Removing log, semver, serde_derive
 exchange: Removing solana
 core: Removing crc, memmap, symlink, untrusted
 perf: Removing serde_derive
 genesis: Removing hex, serde_derive
 sdk-c: Removing sha2
 sys-tuner: Removing semver
 bench-exchange: Removing bincode, bs58, env_logger, serde, serde_derive, untrusted, ws
 btc_spv_bin: Removing serde_json
 btc_spv: Removing chrono
 bpf_loader: Removing serde
 ledger: Removing dlopen, dlopen_derive, serde_derive
 move_loader: Removing byteorder, libc, language_e2e_tests
 ownable: Removing serde, serde_derive
 client: Removing rand
 archiver-utils: Removing rand_chacha
 validator: Removing serde_json, tempfile
 param_passing_dep: Removing solana
 failure: Removing log
 vest: Removing log
 vote-signer: Removing bs58, serde
 local-cluster: Removing symlink
 keygen: Removing rpassword
 install: Removing bs58, log
 upload-perf: Removing log
 runtime: Removing serde_json
 stake: Removing rand

* Add modified Cargo.lock

* fixup! Simplify a few pattern matches

* fixup! Simplify a few pattern matches
2020-02-06 10:02:38 -07:00
Jack May 8436457e75
Rename program_utils.rs (#8127) 2020-02-05 12:48:30 -08:00
Pankaj Garg ed5101b031
Generate max coding shreds when necessary (#8099)
* Generate max coding shreds when necessary

* test
2020-02-04 15:45:01 -08:00
dependabot-preview[bot] c835749563
Bump sys-info from 0.5.8 to 0.5.9 (#8089)
Bumps [sys-info](https://github.com/FillZpp/sys-info-rs) from 0.5.8 to 0.5.9.
- [Release notes](https://github.com/FillZpp/sys-info-rs/releases)
- [Changelog](https://github.com/FillZpp/sys-info-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/FillZpp/sys-info-rs/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-02-03 14:49:39 -07:00
Michael Vines 3bc9789e8d Remove support for 0.22.3 snapshots 2020-01-30 23:34:15 -07:00
carllin 4ffd7693d6
Add lock to make sure slot-based locktree calls are safe (#7993) 2020-01-28 13:45:41 -08:00
Michael Vines fd7d5cbe0d Fix compute_shred_version() (#7989)
automerge
2020-01-27 17:05:31 -08:00
Michael Vines 912aafcefd Reduce epoch duration from 2 weeks to 2 days 2020-01-27 10:34:55 -07:00
Dan Albert c324e71768
Bump cargo toml versions to 0.24.0 (#7976) 2020-01-25 11:04:27 -06:00
Michael Vines 989355e885 Add ability to hard fork at any slot (#7801)
automerge
2020-01-24 17:27:04 -08:00
Michael Vines a2f2c46f87 Ensure shred version is never 0 2020-01-24 17:41:20 -07:00
Michael Vines d62ed4f6b3 Add BlockstoreProcessorResult 2020-01-23 16:52:47 -07:00
Michael Vines b045f9a50d codemod --extensions rs get_snapshot_tar_path get_snapshot_archive_path 2020-01-23 13:37:13 -07:00
Michael Vines ce231602dc Move snapshot archive generation out of the SnapshotPackagerService 2020-01-23 13:37:13 -07:00
Michael Vines 6f5e0cd161 Type grooming 2020-01-23 13:37:13 -07:00
Michael Vines 8b8033c72b Set BankRc slot correctly when restoring a bank snapshot 2020-01-23 13:37:13 -07:00
Michael Vines 7ca0109732 --halt-at-slot 1 now halts at slot 1 2020-01-23 13:37:13 -07:00
Michael Vines 6b5172d002 add_snapshot now returns SlotSnapshotPaths 2020-01-23 13:37:13 -07:00
Michael Vines 934c32cbc6 Add mechanism to load v0.22.3 snapshots on newer Solana versions 2020-01-22 15:40:32 -07:00
Greg Fitzgerald 3aabeb2b81
Rename bootstrap leader (#7906)
* Rename bootstrap leader to bootstrap validator

It's a normal validator as soon as other validators enter the
leader schedule.

* cargo fmt

* Fix build

Thanks @CriesofCarrots!
2020-01-22 09:22:09 -07:00
Sagar Dhawan 37d1daf58e Revert "Generate MAX_DATA_SHREDS_PER_FEC_BLOCK coding shreds for each FEC block (#7474)" (#7898)
automerge
2020-01-21 11:48:09 -08:00
Sagar Dhawan 2dd8ab197d
Remove redundant threadpools in sigverify (#7888)
* Limit the number of thread pools sigverify creates

* Name local threadpools
2020-01-20 20:08:19 -08:00
Tyera Eulberg 21d5fe6272
Fix timestamp overflow (#7886)
* Split timestamp calculation into separate fn for math unit testing

* Add failing test

* Fix failing test; also bump stakes to near expected cluster max supply

* Don't error on timestamp of slot 0
2020-01-20 17:54:44 -07:00
carllin 0526d4ff21
Add logging surrounding failure in `get_slot_entries_with_shred_info()` (#7846)
* Add logging surrounding failure
2020-01-16 16:03:24 -08:00
carllin 76e20015a4
Add separate thread to check for and store duplicate slot proofs (#7834) 2020-01-16 15:27:54 -08:00
Ryo Onodera ed0129f881 Don't depend on unused lazy_static 2020-01-16 08:43:13 -07:00
Tyera Eulberg 6d3b8b6d7d
Remove tuples from JSON RPC responses (#7806)
* Remove RpcConfirmedBlock tuple

* Remove getRecentBlockhash tuple

* Remove getProgramAccounts tuple

* Remove tuple from get_signature_confirmation_status

* Collect Rpc response types

* Camel-case epoch schedule for rpc response

* Remove getBlockCommitment tuple

* Remove getStorageTurn tuple

* Update json-rpc docs
2020-01-15 00:25:45 -07:00
Justin Starry ff1ca1e0d3
Consolidate entry tick verification into one function (#7740)
* Consolidate entry tick verification into one function

* Mark bad slots as dead in blocktree processor

* more feedback

* Add bank.is_complete

* feedback
2020-01-15 09:15:26 +08:00
carllin 721c4378c1
Plumb ability to handle duplicate shreds into shred insertion functions (#7784) 2020-01-14 15:37:53 -08:00
carllin f91ffbbfdf
Add support in BlockStore for tracking duplicate slots (#7761)
* Add test

* Add new column family to track duplicate slots

* Fix clippy errors

* Introduce new SlotColumn for common implementation of Column trait
2020-01-13 17:21:39 -08:00
Pankaj Garg 156292e408
Reduce grace ticks, and ignore grace ticks for missing leaders (#7764)
* Reduce grace ticks, and ignore grace ticks for missing leaders

* address review comments

* blockstore related renames
2020-01-14 05:25:41 +05:30
Greg Fitzgerald b5dba77056 Rename blocktree to blockstore (#7757)
automerge
2020-01-13 13:13:52 -08:00
carllin 8f7ded33e0
coalesce data and coding index (#7765) 2020-01-13 12:03:19 -08:00
Tyera Eulberg a17d5795fb getConfirmedBlock: add encoding optional parameter (#7756)
automerge
2020-01-12 21:34:30 -08:00
carllin fd3c6eb320 Remove print in test (#7758)
automerge
2020-01-10 15:37:22 -08:00
carllin 27d2c0aaf3
Handle errors on replaying ledger properly (#7741) 2020-01-10 12:16:44 -08:00
Ryo Onodera 865c42465a
Cap file size for snapshot data files (#7182)
* save limit deserialize

* save

* Save

* Clean up

* rustfmt

* rustfmt

* Just comment out to please CI

* Fix ci...

* Move code

* Rustfmt

* Crean up control flow

* Add another comment

* Introduce predetermined constant limit on snapshot data files (deserialize side)

* Introduce predetermined constant limit on snapshot data files (serialize side)

* rustfmt

* Tweak message

* Revert dynamic memory limit

* Limit size of snapshot data file (de)serialization

* Fix test breakage

* Clean up

* Fix uses formatting

* Rename: deserialize_{for,from}_snapshot

* Simplify comment

* Use Slot

* Provide slot for status cache

* Align variable name with snapshot_status_cache_file_path

* Define serialize_snapshot_data_file_with_metrics

* Fix build.......

* De-marco serialize_snapshot_data_file_with_metrics

* Revert u64 => Slot
2020-01-10 09:49:36 +09:00
carllin 85c3d64f29
Fix rooted slot iterator (#7695)
* Enable jumping gaps caused by snapshots in rooted slot iterator
2020-01-07 22:51:28 -05:00
Tyera Eulberg e75a64a8a2
getBlockTime: Fix RootedSlotIterator lowest root (#7681)
* Determine lowest_nonzero_root for purged blocktrees, and clean up slot offset math

* Filter duplicate timestamp votes

* Refactor deduping code
2020-01-05 23:38:27 -07:00
dependabot-preview[bot] bc71e1b612 Bump sha2 from 0.8.0 to 0.8.1
Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.8.0 to 0.8.1.
- [Release notes](https://github.com/RustCrypto/hashes/releases)
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.8.0...sha2-v0.8.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-01-05 22:33:02 -07:00
Tyera Eulberg 84a37a2c0c Make validator timestamping more coincident, and increase timestamp sample range (#7673)
automerge
2020-01-03 22:38:00 -08:00
Michael Vines 7002ccb866 Log root slots while processing ledger 2020-01-03 13:25:37 -07:00
Michael Vines 4fe0b116ae Measure heap usage while processing the ledger 2020-01-03 13:25:37 -07:00
Rob Walker 6b7d9942a7
Add authorized_voter history (#7643)
* Add authorized_voter history

* fixups

* coverage

* bigger vote state
2019-12-30 19:57:53 -08:00
Pankaj Garg 87b2525e03
Limit maximum number of shreds in a slot to 32K (#7584)
* Limit maximum number of shreds in a slot to 32K

* mark dead slot replay as fatal error
2019-12-30 07:42:09 -08:00
Parth 727be309b2 fix entryverification state (#7169)
automerge
2019-12-23 23:26:27 -08:00
Michael Vines 9f65d22909
Groom log messages (#7610) 2019-12-23 10:43:07 -07:00
Dan Albert ca10cf081f
Update cargo.toml files from 0.22.0 to 0.23.0 (#7596) 2019-12-20 21:45:42 -05:00
Sagar Dhawan 6a9005645a
Update "limit-ledger-size" to use DeleteRange for much faster deletes (#7515)
* Update "limit-ledger-size" to use DeleteRange for much faster deletes

* Update core/src/ledger_cleanup_service.rs

Co-Authored-By: Michael Vines <mvines@gmail.com>

* Rewrite more idiomatically

* Move max_ledger_slots to a fn for clippy

* Remove unused import

* Detect when all columns have been purged and fix a bug in deletion

* Check that more than 1 column is actually deleted

* Add helper to test that ledger meets minimum slot bounds

* Remove manual batching of deletes

* Refactor to keep some N slots older than the highest root

* Define MAX_LEDGER_SLOTS that ledger_cleanup_service will try to keep around

* Refactor compact range
2019-12-18 11:50:09 -08:00
Tyera Eulberg 6aaf742dfe Extend getConfirmedBlock rpc to return account pre- and post-balances (#7543)
automerge
2019-12-18 09:56:29 -08:00
Sunny Gleason 323673c3c0
Add `compact_cf` calls to reclaim storage during ledger slot purge (#7264) 2019-12-18 10:29:46 -05:00
Sunny Gleason 84b07c81fd
Fix/remove short circuit logic in ledger_purge_batch (&& -> &), found/fixed by @sdhawan (#7526) 2019-12-17 15:08:40 -05:00
anatoly yakovenko 97589f77f8 Pipeline broadcast socket transmit and blocktree record (#7481)
automerge
2019-12-16 17:11:18 -08:00
dependabot-preview[bot] 7ad64c8d45 Bump serde from 1.0.103 to 1.0.104
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.103 to 1.0.104.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.103...v1.0.104)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-12-16 08:50:09 -07:00
Tyera Eulberg 6b88da2b82
Hook up getBlockTime rpc to real data (#7476)
* Add blocktree timestamp helper functions and tests

* Flesh out blocktree::get_block_time

* Move stakes up into rpc to make testing easier; expand tests

* Review comments
2019-12-14 12:23:02 -07:00
Rob Walker f05860672c
Add create-address-with-seed to solana-cli (#7472)
* Add create account with seed to cli

* clippy

* fixup
2019-12-14 04:38:24 -08:00
anatoly yakovenko 27c8df6140 benches (#7480)
automerge
2019-12-13 20:27:11 -08:00
Pankaj Garg 7adc721d96
Generate MAX_DATA_SHREDS_PER_FEC_BLOCK coding shreds for each FEC block (#7474)
* Generate MAX_DATA_SHREDS_PER_FEC_BLOCK coding shreds for each FEC block

* fix tests, and detangle ErasureMeta::status
2019-12-13 14:50:22 -08:00
carllin b7d6ff6770
Fix Blocktree Config (#7399) 2019-12-12 23:27:24 -08:00
Pankaj Garg 75d1aa5135 Perform erasure recovery when repaired data shreds are received (#7463)
automerge
2019-12-12 17:50:28 -08:00
Pankaj Garg d94041e98d
Allow coding shred index to be different than data shred index (#7438)
* Allow coding shred index to be different than data shred index

* move fec_set_index to shred's common header

* fix bench
2019-12-12 16:50:29 -08:00