Commit Graph

762 Commits

Author SHA1 Message Date
Tyera Eulberg 1a13d22984
Fixup iterator method (#16357) 2021-04-04 23:32:51 +00:00
carllin 4e5ef6bce2
Add cluster state verifier logging (#16330)
* Add cluster state verifier logging

* Add duplicate-slots iterator to ledger tool
2021-04-02 21:48:44 -07:00
Tyera Eulberg da27acabcc
Rpc: enable getConfirmedSignaturesForAddress2 to return confirmed (not yet finalized) data (#16281)
* Update blockstore method to allow return of unfinalized signature

* Support confirmed sigs in getConfirmedSignaturesForAddress2

* Add deprecated comments

* Update docs

* Enable confirmed transaction-history in cli

* Return real confirmation_status; fill in not-yet-finalized block time if possible
2021-04-01 04:35:57 +00:00
Tyera Eulberg 18bd47dbe1
Rpc: fix getConfirmedTransaction slot (#16288)
* Fix transaction blockstore apis

* Update blockstore apis in rpc
2021-03-31 21:04:00 -06:00
behzad nouri 3f63ed9a72
removes OrderedIterator and transaction batch iteration order (#16153)
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817

However load_and_execute_transactions is iterating over
  lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.

If iteratorion_order is [1, 2, 0], and i is 0, then:
  lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
  transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).

This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.
2021-03-31 23:59:19 +00:00
Tyera Eulberg 433f1ead1c
Rpc: enable getConfirmedBlock and getConfirmedTransaction to return confirmed (not yet finalized) data (#16142)
* Add Blockstore block and tx apis that allow unrooted responses

* Add TransactionStatusMessage, and send on bank freeze; also refactor TransactionStatusSender

* Track highest slot with tx-status writes complete

* Rename and unpub fn

* Add commitment to GetConfirmed input configs

* Support confirmed blocks in getConfirmedBlock

* Support confirmed txs in getConfirmedTransaction

* Update sigs-for-addr2 comment

* Enable confirmed block in cli

* Enable confirmed transaction in cli

* Review comments

* Rename blockstore method
2021-03-26 16:47:35 -06:00
Jeff Washington (jwash) 4f4cffbd03
Throttle PoH ticks by cumulative slot time (#16139)
* Throttle PoH ticks by cumulative slot time

* respond to pr feedback

* saturating sub

* updated comment
2021-03-26 18:54:16 +00:00
Justin Starry e7fd7d46cf
rpc: add getSlotLeaders method (#16057) 2021-03-23 17:48:54 +00:00
behzad nouri 4f82b897bc
buffers data shreds to make larger erasure coded sets (#15849)
Broadcast stage batches up to 8 entries:
https://github.com/solana-labs/solana/blob/79280b304/core/src/broadcast_stage/broadcast_utils.rs#L26-L29
which will be serialized into some number of shreds and chunked into FEC
sets of at most 32 shreds each:
https://github.com/solana-labs/solana/blob/79280b304/ledger/src/shred.rs#L576-L597
So depending on the size of entries, FEC sets can be small, which may
aggravate loss rate.
For example 16 FEC sets of 2:2 data/code shreds each have higher loss
rate than one 32:32 set.

This commit broadcasts data shreds immediately, but also buffers them
until it has a batch of 32 data shreds, at which point 32 coding shreds
are generated and broadcasted.
2021-03-23 14:52:38 +00:00
carllin d76ad33597
Handle blockstore insert dup checks (#16051) 2021-03-22 16:18:22 -07:00
Tyera Eulberg 654449ce91
Bump tokio to 1.1 (#15926) 2021-03-16 21:32:50 +00:00
behzad nouri 3b85cbc504
renames is_last_in_fec_set back to is_last_data (#15848)
https://github.com/solana-labs/solana/pull/10095
renamed is_last_data to is_last_in_fec_set. However, the code shows that
this is actually meant to indicate where the serialized data is
complete:
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L599-L600
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L229-L231

There are multiple FEC sets for each `&[Entry]` serialized and this flag
does not represent shreds last in FEC sets (only the very last one by
overlap). So the name is wrong and confusing
2021-03-16 10:09:16 +00:00
Michael Vines a2eb655322 =1.7.0 2021-03-16 07:51:07 +00:00
Michael Vines 0c9ca5522c Bump version to v1.7.0 2021-03-13 09:01:21 +00:00
Justin Starry 918d04e3f0
Add more slot update notifications (#15734)
* Add more slot update notifications

* fix merge

* Address feedback and add integration test

* switch to datapoint

* remove unused shred method

* fix clippy

* new thread for rpc completed slots

* remove extra constant

* fixes

* rely on channel closing

* fix check
2021-03-12 21:44:06 +08:00
Trent Nelson 24d18b3cf2 docs: add docs links for crates published to crates.io 2021-03-11 08:38:18 +00:00
Jeff Washington (jwash) 1135ffd595
mut data refs as slice (#15782) 2021-03-10 15:28:03 -06:00
Jeff Washington (jwash) 8a3135d17b
Account->AccountSharedData (#15691) 2021-03-09 15:06:07 -06:00
Jeff Washington (jwash) 9448540ea6
log timing after ledger process (#15716) 2021-03-05 11:01:52 -06:00
Tyera Eulberg 7e65289729
Convert blockstore TransactionStatus column family to protobufs (#15733)
* Prevent panic if TransactionStatus can't be deserialized

* Convert Blockstore TransactionStatus column to protobuf

* Add compatability test
2021-03-05 09:05:35 -07:00
Jeff Washington (jwash) be35c1c1b7
add execute detail timings (#15638) 2021-03-03 17:07:45 -06:00
sakridge f1223fb783
Lower blockstore processor error severity (#15578) 2021-03-01 14:57:37 -08:00
Trent Nelson ef63ffce7d Bump ed25519-dalek to 1.0.1 2021-03-01 21:10:46 +00:00
Michael Vines 33eaa2b238 Sort forks in "ledger processed..." log message 2021-02-28 17:32:50 -08:00
Michael Vines e92cff3efa create-snapshot subcommad now accepts the ROOT keyword 2021-02-26 16:40:10 -08:00
carllin c2e8814dce
Add limit and shrink policy for recycler (#15320) 2021-02-24 00:15:58 -08:00
Michael Vines 5df36aec7d Pacify clippy 2021-02-19 20:08:41 -08:00
Tyera Eulberg 170cb792eb
Return blockstore error if previous_blockhash cannot be determined (#15382)
* Return blockstore error if previous_blockhash cannot be determined

* Add require_previous_blockshash flag
2021-02-18 01:04:52 +00:00
Trent Nelson 7f7370c306 Re-allow clippy::integer_arithmetic at crate-level 2021-02-17 13:55:08 -07:00
Michael Vines 9ba69a7381 Adapt to fs_extra 1.2.0 2021-02-17 12:44:58 -08:00
Tyera Eulberg 98e3e570d2 Add --force arg for bigtable upload 2021-02-16 17:24:07 -08:00
behzad nouri e403aeaf05
adds an inverted index to leader schedule (#15249)
next_leader_slot is doing a linear search for slots in which a pubkey is
the leader:
https://github.com/solana-labs/solana/blob/e59a24d9f/ledger/src/leader_schedule_cache.rs#L123-L157
This can be done more efficiently by adding an inverted index to leader
schedule.
2021-02-15 00:52:52 +00:00
Tyera Eulberg 0812931c38
Log if unsanitary transactions are read from blockstore (#15319) 2021-02-14 06:32:43 +00:00
Tyera Eulberg da6753b8c0
Warp timestamp and extend max-allowable-drift for accommodate slow blocks (#15204)
* Remove timestamp_correction feature gating

* Remove timestamp_bounding feature gating

* Remove unused deprecated ledger code

* Remove unused deprecated unbounded-timestamp code

* Enable independent adjustment of fast/slow timestamp bounding

* Update timestamp bounds to 25% fast, 80% slow; warp timestamp

* Update bank hash test

* Add PR number to feature

Co-authored-by: Michael Vines <mvines@gmail.com>

Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-09 15:49:00 -07:00
sakridge bbae23358c
ledger-tool cleanup and additions (#15179)
* Plumb allow-dead-slots to ledger-tool verify

* ledger-tool cleanup and add some useful missing args

Print root slots and how many unrooted past last root.
2021-02-06 17:26:42 -08:00
sakridge 774416a546
Ledger purge printing (#15176) 2021-02-06 15:45:08 -08:00
Ryo Onodera c69027e5aa
chore: bump serde from 1.0.118 to 1.0.122 (#15126)
* chore: bump serde from 1.0.118 to 1.0.122

* Update programs/bpf/Cargo.lock
2021-02-05 19:23:24 +09:00
Tyera Eulberg d1563f0ccd
Bump tonic, prost, tarpc, tokio (#15013)
* Update tonic & prost, and regenerate proto

* Reignore doc code

* Revert pull #14367, but pin tokio to v0.2 for jsonrpc

* Bump backoff and goauth -> and therefore tokio

* Bump tokio in faucet, net-utils

* Bump remaining tokio, plus tarpc
2021-02-05 00:21:53 -07:00
Michael Vines 42bf6dc2ab Revert hard nofile limit back to 500000 2021-02-03 19:43:55 +00:00
Ryo Onodera 8376781ec8
Don't squash caught errors, please (#15046)
* Don't squash caught errors, please

* Update blockstore.rs

* Update blockstore.rs
2021-02-03 14:42:34 +00:00
Trent Nelson 894b412aef docs: bump nofiles recommendations to match maps 2021-02-02 21:59:21 -07:00
dependabot[bot] 1df93fa2be
chore: bump serde from 1.0.112 to 1.0.118 (#14828)
* chore: bump serde from 1.0.112 to 1.0.122

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.112 to 1.0.122.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.122)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* Update frozen_abi digest following serde update

* Revert "chore: bump serde from 1.0.112 to 1.0.122"

This reverts commit a3ef4442a4c985144ae2bd7ceaf8899a7ab8d7c0.

* Revert "[auto-commit] Update all Cargo lock files"

This reverts commit c41c3b005fb1ccade55155302c52cd5736c4b55f.

* chore: bump serde from 1.0.112 to 1.0.118

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.112 to 1.0.118.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.118)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* Remove serum-dex pinning

* blind commit!

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-02-02 23:28:16 +09:00
Tyera Eulberg cbb8b79a60
Add validator flag to opt in to cpi and logs storage (#14922)
* Add validator flag to opt in to cpi and logs storage

* Default TestValidator to opt-in; allow using in multinode-demo

* No clone

Co-authored-by: Carl Lin <carl@solana.com>
2021-02-01 14:00:51 -07:00
Michael Vines cbffab7850 Upgrade to Rust v1.49.0 2021-01-23 19:16:36 -08:00
sakridge 191193289f
Upgrade sha2 to 0.9.3 (#14746) 2021-01-22 22:25:22 -08:00
Josh 1de6d28eaf
Add block_time to getConfirmedSignaturesForAddress2 and getConfirmedTransaction (#14572)
* add block_time to get_confirmed_signatures_for_address2 and protobuf implementation for tx_by_addr

* add tests for convert

* update cargo lock

* run cargo format after rebase

* introduce legacy TransactionByAddrInfo

* move LegacyTransactionByAddrInfo back to storage-bigtable
2021-01-20 22:10:35 -08:00
dependabot[bot] 4ecc0f4fa6
chore: bump rayon from 1.4.1 to 1.5.0 (#14177)
* chore: bump rayon from 1.4.1 to 1.5.0

Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.4.1 to 1.5.0.
- [Release notes](https://github.com/rayon-rs/rayon/releases)
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md)
- [Commits](https://github.com/rayon-rs/rayon/compare/rayon-core-v1.4.1...rayon-core-v1.5.0)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-01-15 18:07:35 +09:00
sakridge 907f518f6d
Add load/execute/store timings (#14561) 2021-01-14 14:14:16 -08:00
Ryo Onodera e95ebcf864
Don't stop to find newer cluster-confirmed roots (#14557)
* Don't stop to find newer cluster-confirmed roots

* Fix and add new tests

* nits
2021-01-13 15:47:06 +00:00
carllin 6dfad0652f
Cache account stores, flush from AccountsBackgroundService (#13140) 2021-01-11 17:00:23 -08:00
sakridge d8105bb7d7
Add rocskdb high priority threads (#14515)
Without them, memtable writes can stall on compactions.
2021-01-11 13:32:02 -08:00
Michael Vines 86c81a0ba2 Humanize the 'ledger processed...' time 2021-01-08 22:56:21 -08:00
carllin 5affd8aa72
Add secondary indexes (#14212) 2020-12-31 18:06:03 -08:00
Michael Vines 3d077fb656 Revert "Upgrade in-tree tokio 0.2 usage to tokio 0.3"
This reverts commit 444ed768dc.
2020-12-31 06:26:53 +00:00
sakridge 2074e407cd
Add poh speed check and tick speed calibration (#14292) 2020-12-29 09:35:57 -08:00
Michael Vines 444ed768dc Upgrade in-tree tokio 0.2 usage to tokio 0.3 2020-12-29 09:23:23 -08:00
Trent Nelson 5b903318b2 vote: Add helper for creating current-versioned states 2020-12-22 19:37:26 -07:00
sakridge baa9602411
Add shrink paths (#14238) 2020-12-21 21:33:37 -08:00
behzad nouri 6a3797e164
adds crds-value for broadcasting duplicate shreds through gossip (#14133)
In gossip, the header overhead we get from:
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/cluster_info.rs#L434-L435
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/crds_value.rs#L31-L36
https://github.com/solana-labs/solana/blob/de9ac43eb/core/src/crds_value.rs#L73
already exceeds SIZE_OF_NONCE in shreds. We also need aditional
meta-data (wallclock, source pubkey, ...). Which means that given the
SHRED_PAYLOAD_SIZE, we cannot fit all these in PACKET_DATA_SIZE:
https://github.com/solana-labs/solana/blob/de9ac43eb/ledger/src/shred.rs#L80

On top of that, we need 2 shred payloads as the proof of duplicate. So
each DuplicateShred crds value includes only a chunk of the payload,
along with the meta-data to reconstruct the full payload from the chunks
on the receiving end.
2020-12-18 14:32:43 +00:00
behzad nouri d6d76219b6
caches staked nodes computed from vote-accounts (#13929) 2020-12-17 21:22:50 +00:00
Michael Vines e1f2d384db Don't require increased open file limit at ledger creation
Follow-up to 0b92720fdb, `create_new_ledger()` does not require a higher fd limit
2020-12-17 08:46:13 -08:00
Michael Vines 0b92720fdb Don't require increased open file limit in solana-test-validator
Travis CI in particular does not allow the open file limit to be
increased.
2020-12-16 20:43:08 -08:00
Tyera Eulberg ac0d32bc7e
Add blockstore skipped api (#14145)
* Add blockstore api to determine if a slot was skipped

* Return custom rpc error if slot is skipped
2020-12-16 13:40:36 -07:00
Michael Vines 501fd83afd Fix up upgradeable bpf loader activation 2020-12-16 06:36:14 +00:00
sakridge d4a174fb7c
Partial shred deserialize cleanup and shred type differentiation (#14094)
* Partial shred deserialize cleanup and shred type differentiation in retransmit

* consolidate packet hashing logic
2020-12-15 16:50:40 -08:00
Jack May 582418de5e
Cleanup ledger builtins (#14083) 2020-12-15 12:23:49 -08:00
Michael Vines 1e977ac025 Bump version to v1.6.0 2020-12-15 18:28:04 +00:00
Jack May 9e90394583
Upgradeable loader (#13689) 2020-12-14 15:35:10 -08:00
Michael Vines 7143aaa89b Clippy 2020-12-14 08:03:29 -08:00
dependabot[bot] 88d950e5a8
chore: bump log from 0.4.8 to 0.4.11 (#13691)
* chore: bump log from 0.4.8 to 0.4.11

Bumps [log](https://github.com/rust-lang/log) from 0.4.8 to 0.4.11.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/compare/0.4.8...0.4.11)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-12-13 13:48:23 +09:00
dependabot[bot] 8df9bcc784
chore: bump libc from 0.2.80 to 0.2.81 (#14092)
* chore: bump libc from 0.2.80 to 0.2.81

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.80 to 0.2.81.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.80...0.2.81)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-12-13 12:13:17 +09:00
Michael Vines 0a9ff1dc9d Initial solana-test-validator command-line program 2020-12-11 04:17:38 +00:00
Josh 13db3eca9f
SPL token balance in transaction metadata (#13673)
* feat: store pre / post token balances

* move helper functions into separate include

* move token balance functionality to transaction-status crate

* fix blockstore processor test

* fix bigtable legacy test

* add caching to decimals
2020-12-10 19:25:07 -08:00
sakridge aa2751e614
Check shred type in is_duplicate (#14050) 2020-12-10 18:20:08 -08:00
sakridge c5fe076432
Better dupe detection (#13992) 2020-12-09 23:14:31 -08:00
Alexander Meißner a706706572
Validator CLI option to enable just-in-time compilation of BPF (#13789)
* Adds a CLI option to the validator to enable just-in-time compilation of BPF.

* Refactoring to use bpf_loader_program instead of feature_set to pass JIT flag from the validator CLI to the executor.
2020-12-07 09:49:55 +01:00
carllin 51d8f36dae
Ledger-tool recreate snapshot of starting snapshots
Co-authored-by: Carl Lin <carl@solana.com>
2020-12-04 21:14:59 -08:00
behzad nouri 4e8565253c
verifies that serialized size of transactions does not exceed packet data size (#13945) 2020-12-03 22:24:32 +00:00
Tyera Eulberg 85aa632be0
Blockstore exact purge cleanup (#13946)
* Add failing test

* Clean up signature get
2020-12-03 21:49:07 +00:00
Trent Nelson 404fc1570d runtime: Replace `HashAgeKind` with `NonceRollbackInfo` 2020-12-02 20:10:08 +00:00
behzad nouri e1793e5a13
caches vote-state de-serialized from vote accounts (#13795)
Gossip and other places repeatedly de-serialize vote-state stored in
vote accounts. Ideally the first de-serialization should cache the
result.

This commit adds new VoteAccount type which lazily de-serializes
VoteState from Account data and caches the result internally.

Serialize and Deserialize traits are manually implemented to match
existing code. So, despite changes to frozen_abi, this commit should be
backward compatible.
2020-11-30 17:18:33 +00:00
Michael Vines b5f7e39be8 TestValidator public interface cleanup 2020-11-25 17:04:37 -08:00
Ryo Onodera b74d7b5758
Fix fragile tests in prep of stake rewrite pr (#13654)
* Fix fragile tests in prep of stake rewrite pr

* Restore BOOTSTRAP_VALIDATOR_LAMPORTS where appropriate

* Further clean up

* Further clean up

* Aligh with other call site change

* Remove false warn!

* fix ci!
2020-11-20 17:21:03 +09:00
carllin afc1b59475
Fix assertion failure (#13626)
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-16 21:30:38 -08:00
dependabot[bot] fc54fb88ae chore: bump futures from 0.3.5 to 0.3.8
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.5 to 0.3.8.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.5...0.3.8)

Signed-off-by: dependabot[bot] <support@github.com>
2020-11-14 08:31:19 -08:00
Justin Starry 01a4889b53 clippy 2020-11-13 07:21:38 -08:00
Justin Starry d611337394 Fix overflow in entry hash count verification 2020-11-13 07:21:38 -08:00
Ryo Onodera 89b474e192
Fix slow/stuck unstaking due to toggling in epoch (#13501)
* Fix slow/stuck unstaking due to toggling in epoch

* nits

* nits

* Add stake_program_v2 feature status check to cli

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-11-11 14:11:57 -07:00
sakridge b4cf968e14
Add back shredding broadcast stats (#13463) 2020-11-09 23:04:27 -08:00
Tyera Eulberg bc62313c66
Allow feature builtins to overwrite existing builtins (#13403)
* Allow feature builtins to overwrite existing builtins

* Add feature_builtin ActivationType

* Correctly retain idempotent for replacing case

* Fix test

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-11-05 15:37:07 +00:00
Tyera Eulberg 80db6c0980 Add bounded timestamp-estimation method 2020-10-31 21:30:42 -07:00
Michael Vines 225bed11c7 Remove Entrypoint type 2020-10-29 21:45:24 -07:00
Michael Vines df8dab9d2b Native/builtin programs now receive an InvokeContext 2020-10-29 21:45:24 -07:00
Jack May c458d4b213
move Account to solana-sdk (#13198) 2020-10-28 22:01:07 -07:00
Tyera Eulberg 39686ef098
Use bank timestamp to populate Blockstore::blocktime_cf when correction active (#13158) 2020-10-26 19:23:45 +00:00
Michael Vines 959880db60 Remove unused pubkey::Pubkey imports 2020-10-21 19:08:13 -07:00
Michael Vines 7bc073defe Run `codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand` 2020-10-21 19:08:13 -07:00
Ryo Onodera 0776fa05c7
Add ledger-tool dead-slots and improve purge a lot (#13065)
* Add ledger-tool dead-slots and improve purge a lot

* Reduce batch size...

* Add --dead-slots-only and fixed purge ordering
2020-10-21 17:45:21 +00:00
behzad nouri 75d62ca095
improves threads' utilization in processing gossip packets (#12962)
ClusterInfo::process_packets handles incoming packets in a thread_pool:
https://github.com/solana-labs/solana/blob/87311cce7/core/src/cluster_info.rs#L2118-L2134

However, profiling runtime shows that threads are not well utilized and
a lot of the processing is done sequentially.

This commit redistributes the work done in parallel. Testing on a gce
cluster shows 20%+ improvement in processing gossip packets with much
smaller variations.
2020-10-19 19:03:38 +00:00
Tyera Eulberg 359707c85e
Convert Blockstore Rewards cf to protobuf (#12860)
* Add Blockstore protobuf cf type

* Add Rewards message to proto and make generated pub

* Convert Rewards cf to ProtobufColumn

* Add bench

* Adjust tags

* Move solana proto definitions and conversion methods to new crate
2020-10-15 18:04:10 -06:00
Michael Vines 8a119c1483 Bump reqwest/rayon to get past `cargo audit` 2020-10-11 18:51:09 +00:00
Tyera Eulberg b028c47d2b
Correct Bank timestamp drift every slot (#12737)
* Move timestamp helper to sdk

* Add Bank method for getting timestamp estimate

* Return sysvar info from Bank::clock

* Add feature-gated timestamp correction

* Rename unix_timestamp method to be more descriptive

* Review comments

* Add timestamp metric
2020-10-09 21:53:41 +00:00
Ryo Onodera 1f4bcf70b0
Fix various ledger-tool error due to no builtins (#12759)
* Fix various ledger-tool error due to no builtins

* Add missing file...
2020-10-09 12:19:36 -06:00
Josh 8f5431551e
Store program logs in blockstore / bigtable (TransactionWithStatusMeta) (#12678)
* introduce store program logs in blockstore / bigtable

* fix test, transaction logs created for successful transactions

* fix test for legacy bincode implementation around log_messages

* only api nodes should record logs

* truncate transaction logs to 100KB

* refactor log truncate for improved coverage
2020-10-08 12:06:15 -07:00
Michael Vines 11df2e2236 Bump version to v1.5.0 2020-10-08 04:51:36 +00:00
sakridge 71a308affd
Use datapoint instead of print for is_full signal (#12673) 2020-10-07 13:36:54 -07:00
sakridge ce98088457
Track inserted repair shreds (#12455) 2020-09-29 14:13:21 -07:00
Michael Vines 31696a1d72 Port BPFLoader2 activation to FeatureSet and rework built-in program activation 2020-09-28 12:50:19 -07:00
dependabot[bot] 0718501c72 chore: bump rocksdb from 0.14.0 to 0.15.0
Bumps [rocksdb](https://github.com/rust-rocksdb/rust-rocksdb) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/rust-rocksdb/rust-rocksdb/releases)
- [Changelog](https://github.com/rust-rocksdb/rust-rocksdb/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-rocksdb/rust-rocksdb/compare/v0.14.0...v0.15.0)

Signed-off-by: dependabot[bot] <support@github.com>
2020-09-28 09:52:31 -07:00
Michael Vines c10da16d7b Port instructions sysvar and secp256k1 program activation to FeatureSet 2020-09-25 11:40:36 -07:00
Justin Starry 6601ec8f26
Record and store invoked instructions in transaction meta (#12311)
* Record invoked instructions and store in transaction meta

* Enable cpi recording if transaction sender is some

* Rename invoked to innerInstructions
2020-09-24 22:36:22 +08:00
Justin Starry 731a943239
Remove transaction encoding from storage layer (#12404) 2020-09-24 13:10:29 +08:00
sakridge 68e5a2ef56
Add RPC notify and banking keys debug (#12396) 2020-09-23 18:46:42 -07:00
Josh 65a6bfad09
Add blockstore column to store performance sampling data (#12251)
* Add blockstore column to store performance sampling data

* introduce timer and write performance metrics to blockstore

* introduce getRecentPerformanceSamples rpc

* only run on rpc nodes enabled with transaction history

* add unit tests for get_recent_performance_samples

* remove RpcResponse from rpc call

* refactor to use Instant::now and elapsed for timer

* switch to root bank and ensure not negative subraction

* Add PerfSamples to purge/compaction

* refactor to use Instant::now and elapsed for timer

* switch to root bank and ensure not negative subraction

* remove duplicate constants

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-09-22 12:26:32 -07:00
Ryo Onodera cb8661bd49
Persistent tower (#10718)
* Save/restore Tower

* Avoid unwrap()

* Rebase cleanups

* Forcibly pass test

* Correct reconcilation of votes after validator resume

* d b g

* Add more tests

* fsync and fix test

* Add test

* Fix fmt

* Debug

* Fix tests...

* save

* Clarify error message and code cleaning around it

* Move most of code out of tower save hot codepath

* Proper comment for the lack of fsync on tower

* Clean up

* Clean up

* Simpler type alias

* Manage tower-restored ancestor slots without banks

* Add comment

* Extract long code blocks...

* Add comment

* Simplify returned tuple...

* Tweak too aggresive log

* Fix typo...

* Add test

* Update comment

* Improve test to require non-empty stray restored slots

* Measure tower save and dump all tower contents

* Log adjust and add threshold related assertions

* cleanup adjust

* Properly lower stray restored slots priority...

* Rust fmt

* Fix test....

* Clarify comments a bit and add TowerError::TooNew

* Further clean-up arround TowerError

* Truly create ancestors by excluding last vote slot

* Add comment for stray_restored_slots

* Add comment for stray_restored_slots

* Use BTreeSet

* Consider root_slot into post-replay adjustment

* Tweak logging

* Add test for stray_restored_ancestors

* Reorder some code

* Better names for unit tests

* Add frozen_abi to SavedTower

* Fold long lines

* Tweak stray ancestors and too old slot history

* Re-adjust error conditon of too old slot history

* Test normal ancestors is checked before stray ones

* Fix conflict, update tests, adjust behavior a bit

* Fix test

* Address review comments

* Last touch!

* Immediately after creating cleaning pr

* Revert stray slots

* Revert comment...

* Report error as metrics

* Revert not to panic! and ignore unfixable test...

* Normalize lockouts.root_slot more strictly

* Add comments for panic! and more assertions

* Proper initialize root without vote account

* Clarify code and comments based on review feedback

* Fix rebase

* Further simplify based on assured tower root

* Reorder code for more readability

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-19 14:03:54 +09:00
carllin 3533e11786
Fix blockstore processor squash (#12319) 2020-09-17 21:33:08 -07:00
sakridge 3930cb865a
Add keccak-secp256k1 instruction (#11839)
* Implement keccak-secp256k1 instruction

Verifies eth addreses with ecrecover function

* Move secp256k1 test
2020-09-15 18:23:21 -07:00
Ryo Onodera de4a613610
Check bank capitalization (#11927)
* Check bank capitalization

* Simplify and unify capitalization calculation

* Improve and add tests

* Avoid overflow and inhibit automatic restart

* Fix test

* Tweak checked sum for cap. and add tests

* Fix broken build after merge conflicts..

* Rename to ClusterType

* Rename confusing method

* Clarify comment

* Verify cap. in rent and inflation tests

Co-authored-by: Stephen Akridge <sakridge@gmail.com>
2020-09-12 01:48:06 +09:00
Tyera Eulberg 05db41fe9c
Cache block time in Blockstore (#11955)
* Add blockstore column to cache block times

* Add method to cache block time

* Add service to cache block time

* Update rpc getBlockTime to use new method, and refactor blockstore slightly

* Return block_time with confirmed block, if available

* Add measure and warning to cache-block-time
2020-09-09 09:33:14 -06:00
Ryo Onodera c274e26eb8
Prevent unbound memory growth by blockstore_processor (#12110)
* Prevent unbound memory growth by blockstore_processor

* Promote log to info! considering infrequency

* Exclude the time of freeing from interval...

* Skip not-shrinkable slots even if forced

* Add comment
2020-09-09 12:05:10 +09:00
Ryo Onodera 53b8ea4464
Rename to ClusterType and restore devnet compat. (#12068)
* Rename to ClusterType and restore devnet compat.

* De-duplicate parse code and add comments

* Adjust default Devnet genesis & reduce it in tests
2020-09-08 23:55:09 +09:00
Michael Vines cd3c134b58 Remove dead code 2020-09-04 16:01:49 -07:00
Michael Vines d3611f74c8 Add exit flag for bigtable upload operations 2020-09-04 16:01:49 -07:00
Michael Vines bafdcf24f5 Add BigTableUploadService 2020-09-04 16:01:49 -07:00
Michael Vines bc7731b969 Add BigTableUploadService 2020-09-04 16:01:49 -07:00
Michael Vines 91a56caed2 Relocate BigTable uploader to ledger/ crate 2020-09-04 16:01:49 -07:00
carllin 1c1a3f979d
Detect and notify when deserializable shreds are available (#11816)
* Add logic to check for complete data ranges

* Add RPC signature notification

Co-authored-by: Carl <carl@solana.com>
2020-09-01 22:06:06 -07:00
sakridge c4253dc0f9
Update to rayon 1.4.0 (#11898) 2020-08-28 21:00:11 -07:00
Jack May db4bbb3569
Switch programs activation to whole-set based gating (#11750)
* Implement Debug for MessageProcessor

* Switch from delta-based gating to whole-set gating

* Remove dbg!

* Fix clippy

* Clippy

* Add test

* add loader to stable operating mode at proper epoch

* refresh_programs_and_inflation after ancestor setup

* Callback via snapshot; avoid account re-add; Debug

* Fix test

* Fix test and fix the past history

* Make callback management stricter and cleaner

* Fix test

* Test overwrite and frozen for native programs

* Test epoch callback with genesis-programs

* Add assertions for parent bank

* Add tests and some minor cleaning

* Remove unsteady assertion...

* Fix test...

* Fix DOS

* Skip ensuring account by dual (whole/delta) gating

* Fix frozen abi implementation...

* Move compute budget constatnt init back into bank

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-08-26 01:49:15 +09:00
Michael Vines 247f27af37
Submit a vote timestamp every vote (#10630)
* Submit a timestamp for every vote

* Submit at most one vote timestamp per second

* Submit a timestamp for every new vote

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-08-21 11:10:51 -06:00
carllin f7adb68599
Squash supermajority root on blockstore replay at startup (#11727) 2020-08-20 21:56:25 -07:00
Trent Nelson 8841c3398c Bump MacOS nofile recommendation message 2020-08-20 23:57:07 +00:00
carllin b7ed06b17a
Cleanup test utilities (#11723)
* Add voting utility

* Add blockstore utility

Co-authored-by: Carl <carl@solana.com>
2020-08-20 05:04:38 +00:00
Trent Nelson 11951eb009 Sync FD limit and max maps to 500k 2020-08-19 16:36:03 +00:00
Michael Vines d15173ad9d Address latest nightly clippy lints, but globally disable stable_sort_primitive 2020-08-17 22:36:10 -07:00
Michael Vines adc984a225 Rename Binary64 to Base64. Establish Base58 encoding 2020-08-17 17:26:29 -07:00
Michael Vines b5f3ced860 Add base64 (binary64) encoding for getConfirmedTransaction/getConfirmedBlock 2020-08-17 17:26:29 -07:00
Tyera Eulberg 6c5b8f324a
Rpc: Add until parameter for getConfirmedSignaturesForAddress2 (#11644)
* Refactor bigtable apis to accept start and end keys

* Make helper fn to deserialize cell data

* Refactor get_confirmed_signatures_for_address to use get_row_data range

* Add until param to get_confirmed_signatures_for_address

* Add until param to blockstore api

* Plumb until through client/cli

* Simplify client params
2020-08-15 10:42:17 -06:00
Jack May 7c736f71fe
Make BPF Loader static (#11516) 2020-08-14 12:32:45 -07:00
Tyera Eulberg 99fb36fe45
RPC: getConfirmedSignaturesForAddress2 only returns confirmed signatures (#11615)
* Add failing test case

* Limit to only rooted slots
2020-08-13 11:07:42 -06:00
Tyera Eulberg de5fb3ba0e
Blockstore address signatures: handle slots that cross primary indexes, and refactor get_confirmed_signatures_for_address2 (#11497)
* Freeze address-signature index in the middle of slot to show failure case

* Secondary filter on signature

* Use AddressSignatures iterator instead of manually decrementing slots

* Remove unused method

* Add metrics

* Add transaction-status-index doccumentation
2020-08-10 10:27:38 -06:00
carllin 7e25130529
Send votes from banking stage to vote listener (#11434)
*  Send votes from banking stage to vote listener

Co-authored-by: Carl <carl@solana.com>
2020-08-07 11:21:35 -07:00
Tyera Eulberg 5530ee4c95
Add address-based lower bound to get_confirmed_signatures_for_address2 loop (#11426) 2020-08-06 22:29:25 +00:00
Tyera Eulberg 1061b50665
Fix blockstore empty panic (#11423)
* Add panicking test

* Add failing test: fresh transaction-status column shouldn't point at valid root 0

* Prevent transaction status match outside of primary-index bounds

* Initialize transaction-status and address-signature primer entries with Slot::MAX

* Revert "Add failing test: fresh transaction-status column shouldn't point at valid root 0"

This reverts commit cbad2a9fae22e5531e3b4ff1b0a9d6a223826c71.

* Revert "Initialize transaction-status and address-signature primer entries with Slot::MAX"

This reverts commit ffaeac0669d0cbe18dd68b5ce177e15a92360b72.
2020-08-06 22:21:46 +00:00
Greg Fitzgerald 3fdd8ffdf0
Remove circular dep between InstructionError and SystemError (#11427) 2020-08-06 22:04:43 +00:00
Michael Vines 02c0981ecf Rename startAfter to before 2020-08-05 14:33:14 -07:00
Michael Vines a11f137810 Rework get_confirmed_signatures_for_address2 2020-08-05 14:33:14 -07:00
Michael Vines 1b2276520b Add getConfirmedSignaturesForAddress2 RPC method 2020-08-05 14:33:14 -07:00
Michael Vines 494968be66 UiTransactionEncoding is now copy 2020-08-05 14:27:12 -07:00
Trent Nelson e12ab9d0dd Bump version to 1.4.0 2020-08-05 12:04:15 -06:00
Michael Vines eefcf484cb clippy 2020-08-03 18:35:15 +00:00
carllin bf18524368
Add hook for getting vote transactions on replay (#11264)
* Add hook for getting vote transactions on replay

Co-authored-by: Carl <carl@solana.com>
2020-07-29 23:17:40 -07:00
Greg Fitzgerald 8ccce1e543
Upgrade ed25519-dalek (#11183) 2020-07-23 17:23:51 -06:00
Tyera Eulberg 6c38369042
Use OrderedIterator in TransactionStatusService (#11149)
* Split out get-first-err for unit testing

* Add failing test

* Add missing ordering
2020-07-21 20:06:49 +00:00
Tyera Eulberg 9a80e31bae
Expose tss to the other blockstore_processor path (#11070) 2020-07-14 21:14:48 -06:00
dependabot-preview[bot] cee9177ec7 Bump libc from 0.2.71 to 0.2.72
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.71 to 0.2.72.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.71...0.2.72)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-07-13 15:00:27 -07:00
Michael Vines 491f5ae61a Add block time placeholder to getConfirmedBlock 2020-07-10 14:47:11 +00:00
dependabot-preview[bot] 841ecfd927
chore(deps): bump bincode from 1.2.1 to 1.3.1 (#10867)
* chore(deps): bump bincode from 1.2.1 to 1.3.1

Bumps [bincode](https://github.com/servo/bincode) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/servo/bincode/releases)
- [Commits](https://github.com/servo/bincode/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

* Switch from deprecated method

* Add options to maintain behavior with bincode::options()

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-07-09 00:08:05 +00:00
carllin 3f6042d8b3
Add RepairWeight to track votes seen in gossip for weighted repair (#10903)
* Add RepairWeight

Co-authored-by: Carl <carl@solana.com>
2020-07-06 22:49:40 -07:00
sakridge 58a475b789
Add db recovery methods (#10838) 2020-07-06 12:43:45 -07:00
sakridge 658de5b347
Skip and warn for hard-forks which are less than the start slot (#10918)
* Skip and warn for hard-forks which are less than the start slot

Option is used during a restart, but then after the restart is
complete, then the option is not needed if the starting slot
is past the hard-fork since the hard-fork should already be
in the snapshot it started from.

* Update ledger/src/blockstore_processor.rs

Co-authored-by: Michael Vines <mvines@gmail.com>

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-07-05 13:51:43 -07:00
carllin f014c94369
Add breakdown of erasure blobs (#10912)
Co-authored-by: Carl <carl@solana.com>
2020-07-03 17:44:32 -07:00
carllin f17ac70bb2
Add weighted traversal (#10877)
Co-authored-by: Carl <carl@solana.com>
2020-07-02 14:33:04 -07:00
Tyera Eulberg 2669ccb864
Update struct prefixes to Ui (#10874)
* Update comments

* Use Ui prefix
2020-07-01 20:06:40 +00:00
dependabot-preview[bot] 4deec51703
chore(deps): bump rayon from 1.3.0 to 1.3.1 (#10605)
* chore(deps): bump rayon from 1.3.0 to 1.3.1

Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.3.0 to 1.3.1.
- [Release notes](https://github.com/rayon-rs/rayon/releases)
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md)
- [Commits](https://github.com/rayon-rs/rayon/compare/rayon-core-v1.3.0...v1.3.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-07-01 08:09:28 -07:00
Ryo Onodera b89e506cbb
Consistently create temp dirs under ledger/farf (#10848) 2020-07-01 11:38:59 +09:00
sakridge 583cec922b
Remove ledger purge batching (#10830) 2020-06-29 14:44:35 -07:00
carllin 0fde0d7379
Add ancestor iterator to lib.rs (#10813)
Co-authored-by: Carl <carl@solana.com>
2020-06-25 23:39:55 +00:00
Greg Fitzgerald 1c498369b5
Remove fee-payer guesswork from Message and Transaction (#10776)
* Make Message::new_with_payer the default constructor

* Remove Transaction::new_[un]signed_instructions

These guess the fee-payer instead of stating it explicitly
2020-06-24 14:52:38 -06:00
sakridge 0e393a5684
Allow for hard fork at last root (#10762) 2020-06-23 21:49:11 -07:00
sakridge a1ef921b88
Rework backup and clear function (#10751) 2020-06-23 21:29:07 +00:00
carllin 77b8de193c
Add utility functions for testing (#10749)
* Add ancestor iterator

* Add blockstore generation from trees

Co-authored-by: Carl <carl@solana.com>
2020-06-23 12:05:00 -07:00
sakridge 2ba8fc5243
Remove slots past wait-for-supermajority slot. (#10720) 2020-06-22 20:27:25 -07:00
carllin a33fef9af2
Add repair breakdown by slot and index (#10717)
* Slot full logging

* Repair stats logging

Co-authored-by: Carl <carl@solana.com>
2020-06-19 18:28:15 -07:00
Greg Fitzgerald 0550b893b0
Fix typos (#10675)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-06-17 20:54:52 -07:00
Greg Fitzgerald 6ee222363e
Move BankForks to solana_runtime (#10637)
* Move BankForks to solana_runtime

* Update imports
2020-06-17 15:27:03 +00:00
Kristofer Peterson 50c93d4441
Cleanup error type mapping in serde_snapshot (#10580)
* Cleanup error type mapping in serde_snapshot

* Fail when account storage entry files cannot be moved from snapshot
2020-06-17 17:56:29 +09:00
sakridge 0de6c444d6
Simd poh (#10604)
* Simd poh

* Fix poh verify bench
2020-06-16 23:03:26 -07:00
sakridge 1eca9b19ab
Entry verify cleanup and gossip counters (#10632)
* Add prune message counter

* Switch to us verification time to match other counters

* Add separate transaction/poh verify timing
2020-06-16 14:00:29 -07:00
dependabot-preview[bot] 480f69cc5e Bump serde from 1.0.111 to 1.0.112
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.111 to 1.0.112.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.111...v1.0.112)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-06-15 07:58:16 -07:00
R. M. Shea 48febf153b
Change documentation introduction to indicate Foundation IP transfer (#10440) 2020-06-13 14:41:05 -07:00
Greg Fitzgerald 2eb6f498a8
Remove redundant BankForks parameter (#10537) 2020-06-12 11:04:17 -06:00
Kristofer Peterson e23340d89e
Clippy cleanup for all targets and nighly rust (also support 1.44.0) (#10445)
* address warnings from 'rustup run beta cargo clippy --workspace'

minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs

expect some performance improvement AccountsDB::clean_accounts()

* address warnings from 'rustup run beta cargo clippy --workspace --tests'

* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'

* rustfmt

* fix warning stragglers

* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned

* Remove lock around JsonRpcRequestProcessor (#10417)

automerge

* make ancestors parameter optional to avoid forcing construction of empty hash maps

Co-authored-by: Greg Fitzgerald <greg@solana.com>
2020-06-09 09:38:14 +09:00
sakridge 2725acebea
Lower counter level (#10428) 2020-06-05 15:53:09 -07:00
Michael Vines aa6832964c
ledger_cleanup_service: compact at a slower rate than purging (#10414) 2020-06-04 21:06:06 -07:00
dependabot-preview[bot] f901571026 Bump serde from 1.0.110 to 1.0.111
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.110 to 1.0.111.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.110...v1.0.111)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-06-02 23:29:14 -07:00
Ryo Onodera caa7f7a0c9
Support opening an in-use rocksdb as secondary (#10209)
automerge
2020-06-02 21:32:44 -07:00
Tyera Eulberg eee9a08376
Purge TransactionStatus and AddressSignatures exactly from ledger-tool (#10358)
* Add failing test

* Add execution path to purge primary-index columns exactly

* Fail gracefully if older TransactionStatus rocksdb keys are present

* Remove columns_empty check for special columns

* Move blockstore purge methods to submodule

* Remove unused column empty check
2020-06-02 19:49:31 -06:00
Greg Fitzgerald 19d11800bf
Remove WithSigner (#10325)
automerge
2020-05-29 23:17:44 -07:00
carllin bc37dfc4c1
log leader (#10316)
automerge
2020-05-29 04:35:20 -07:00
Kristofer Peterson fb4d8e1f62
cleanup clippy tests (#10172)
automerge
2020-05-29 00:26:06 -07:00
dependabot-preview[bot] a71833c17b
Bump regex from 1.3.8 to 1.3.9 (#10308)
* Bump regex from 1.3.8 to 1.3.9

Bumps [regex](https://github.com/rust-lang/regex) from 1.3.8 to 1.3.9.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.3.8...1.3.9)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-05-28 21:48:39 -07:00
Michael Vines 9eefb2c929 Bump version to 1.3.0 2020-05-28 15:01:13 -07:00
dependabot-preview[bot] 9227874ada
Bump regex from 1.3.7 to 1.3.8 (#10293)
* Bump regex from 1.3.7 to 1.3.8

Bumps [regex](https://github.com/rust-lang/regex) from 1.3.7 to 1.3.8.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/regex-1.3.7...1.3.8)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-05-28 10:02:23 -07:00
Michael Vines 5ac2ae1178
Purge next slots to avoid a blockstore_processor panic on restart (#10281) 2020-05-27 19:55:40 -07:00
Michael Vines e3b444834f
Add more logging while unpacking snapshots (#10266) 2020-05-27 10:41:05 -07:00
Michael Vines 7abdfc23f9
succceed is not a word 2020-05-26 23:17:50 -07:00
dependabot-preview[bot] 1bfc4c1489
Bump libc from 0.2.70 to 0.2.71 (#10249)
* Bump libc from 0.2.70 to 0.2.71

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.70 to 0.2.71.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.70...0.2.71)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2020-05-26 11:04:15 -07:00
Michael Vines 156387aba4
LedgerCleanupService no longer causes an OOM and actually purges (#10199)
* cleanup_ledger() now services new_root_receiver while purging
* purge_slots() now fully deletes before compacting
* Add ledger pruning grafana graph
2020-05-24 21:41:54 -07:00
dependabot-preview[bot] 8a8384e674
Bump sha2 from 0.8.1 to 0.8.2 (#10205)
* Bump sha2 from 0.8.1 to 0.8.2

Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.8.1 to 0.8.2.
- [Release notes](https://github.com/RustCrypto/hashes/releases)
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.8.1...sha2-v0.8.2)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-05-24 19:44:04 -07:00
dependabot-preview[bot] 2d3a906d55
Bump tar from 0.4.27 to 0.4.28 (#10185)
Bumps [tar](https://github.com/alexcrichton/tar-rs) from 0.4.27 to 0.4.28.
- [Release notes](https://github.com/alexcrichton/tar-rs/releases)
- [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.27...0.4.28)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-05-22 15:55:40 -07:00
Kristofer Peterson b7a32f01c0
Multi-version snapshot support (#9980)
* Multi-version snapshot support

* rustfmt

* Remove CLI options and runtime support for selection output snapshot version.
Address some clippy complaints.

* Muzzle clippy type complexity warning.

Despite clippy's suggestion, it is not currently possible to create type aliases
for traits and so everything within the 'Box<...>' cannot be type aliased.

This then leaves creating full blown traits, and either implementing
said traits by closure (somehow) or moving the closures into new structs
implementing said traits which seems a bit of a palaver.

Alternatively it is possible to define and use the type alias 'type ResultBox<T> = Result<Box<T>>'
which does seems rather pointless and not a great reduction in complexity but is enough to keep clippy happy.

In the end I simply went with squelching the clippy warning.

* Remove now unused Serialize/Deserialize trait implementations for AccountStorageEntry and AppendVec

* refactor versioned de/serialisers

* rename serde_utils to serde_snapshot

* move call to accounts_db.generate_index() back down to context_accountsdb_from_stream()

* update version 1.1.1 to 1.2.0
remove nested use of serialize_bytes

* cleanups

* Add back measurement of account storage entry serialization.
Remove construction of Vec and HashMap temporaries during serialization.

* consolidate serialisation test cases into serde_snapshot.
clean up leakage of implementation details in serde_snapshot.

* move short term / legacy snapshot code into child module

* add serialize_iter_as_tuple

* preliminary integration of following commit

commit 6d58b73c47294bfb93465d5a83cd2175660b6e6d
Author: Ryo Onodera <ryoqun@gmail.com>
Date:   Wed May 20 14:02:02 2020 +0900

    Confine snapshot 1.1 relic to versioned codepath

* refactored serde_snapshot, rustfmt
legacy accounts_db format now "owns" both leading u64s, legacy bank_rc format has none

* reduce type complexity (clippy)
2020-05-23 02:54:24 +09:00
dependabot-preview[bot] 8da2e1b2f7
Bump tar from 0.4.26 to 0.4.27 (#10150)
Bumps [tar](https://github.com/alexcrichton/tar-rs) from 0.4.26 to 0.4.27.
- [Release notes](https://github.com/alexcrichton/tar-rs/releases)
- [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.26...0.4.27)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-05-20 20:04:22 -07:00
carllin 439fd30840
Fix erasure (#10095)
* Fix bad FEC blocks

* Add test

Co-authored-by: Carl <carl@solana.com>
2020-05-19 16:13:12 -07:00
carllin 97f2bcff69
master: Add nonce to shreds repairs, add shred data size to header (#10109)
* Add nonce to shreds/repairs

* Add data shred size to header

Co-authored-by: Carl <carl@solana.com>
2020-05-19 12:38:18 -07:00
sakridge f562ed4cc8 Distinguish between shred type in shred fetch stage duplicate filter (#10068)
* Shred type check

* Test
2020-05-15 13:23:56 -07:00
Greg Fitzgerald 5e89bd8868
Panic if no fee-payer found via Message::new() (#10050)
automerge
2020-05-15 12:23:09 -07:00
Michael Vines 7080fb9b37
Abort if the open fd limit cannot be increased (#10064)
automerge
2020-05-15 12:14:21 -07:00
Kristofer Peterson 58ef02f02b
9951 clippy errors in the test suite (#10030)
automerge
2020-05-15 09:35:43 -07:00
dependabot-preview[bot] 6a6c5f196a
Bump libc from 0.2.69 to 0.2.70 (#10006)
* Bump libc from 0.2.69 to 0.2.70

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.69 to 0.2.70.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.69...0.2.70)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-05-15 06:18:43 -06:00
sakridge 9575afc8fa
Refactor blockstore recovery code (#10008) 2020-05-13 10:09:38 -07:00
Ryo Onodera 1eb40c3fe0
Introduce eager rent collection (#9527)
* Switch AccountsIndex.account_maps from HashMap to BTreeMap

* Introduce eager rent collection

* Start to add tests

* Avoid too short eager rent collection cycles

* Add more tests

* Add more tests...

* Refacotr!!!!!!

* Refactoring follow up

* More tiny cleanups

* Don't rewrite 0-lamport accounts to be deterministic

* Refactor a bit

* Do hard fork, restore tests, and perf. mitigation

* Fix build...

* Refactor and add switch over for testnet (TdS)

* Use to_be_bytes

* cleanup

* More tiny cleanup

* Rebase cleanup

* Set Bank::genesis_hash when resuming from snapshot

* Reorder fns and clean ups

* Better naming and commenting

* Yet more naming clarifications

* Make prefix width strictly uniform for 2-base partition_count

* Fix typo...

* Revert cluster-dependent gate

* kick ci?

* kick ci?

* kick ci?
2020-05-13 16:22:14 +09:00
sakridge 903a8a3196
Add retransmit packets_by_slot metrics (#9975) 2020-05-11 13:49:10 -07:00
dependabot-preview[bot] 288c9751c1
Bump serde from 1.0.106 to 1.0.110 (#9964)
* Bump serde from 1.0.106 to 1.0.110

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.106 to 1.0.110.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.106...v1.0.110)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-05-10 16:24:25 -07:00
carllin ad3c8fb812
More logging around failure (#9967)
automerge
2020-05-10 16:01:20 -07:00
dependabot-preview[bot] c970bbea4f
Bump serde_bytes from 0.11.3 to 0.11.4 (#9914)
Bumps [serde_bytes](https://github.com/serde-rs/bytes) from 0.11.3 to 0.11.4.
- [Release notes](https://github.com/serde-rs/bytes/releases)
- [Commits](https://github.com/serde-rs/bytes/compare/0.11.3...0.11.4)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-05-07 23:26:37 -07:00
Trent Nelson f12c6c1ed1
BSD compat for 9493de4 (#9922)
automerge
2020-05-07 16:38:33 -07:00
Michael Vines 09ae61651a
Eliminate BankForksInfo (#9887) 2020-05-06 08:24:59 -07:00
Michael Vines 72312ad615
Avoid holding the entire rooted path while loading bank forks (#9885) 2020-05-05 19:45:41 -07:00
carllin 3442f36f8a
Repair alternate versions of dead slots (#9805)
Co-authored-by: Carl <carl@solana.com>
2020-05-05 14:07:21 -07:00
Ryo Onodera a91236012d
Pass around --max-genesis-archive-unpacked-size (#9161)
automerge
2020-04-29 18:53:34 -07:00
Stephen Akridge eb24f3df84 Update dalek version 2020-04-28 21:02:47 -06:00
Tyera Eulberg a12428a5b8
Use Blockstore lowest_slot to start root iterator (#9738) 2020-04-28 10:22:10 -06:00
Jack May efad193180
Make default programs static (#9717) 2020-04-27 21:05:12 -07:00
Michael Vines c11abf88b7
Clean up `use` to keep rust 1.43.0 from complaining (#9740) 2020-04-27 16:54:11 -07:00
Tyera Eulberg a7f33b5014
Cache banks in BankForks until optional largest_confirmed_root (#9678)
automerge
2020-04-24 15:49:57 -07:00
sakridge fa20963b93
Revert shred fs (#9712)
* Revert "Untar is called for shred archives that do not exist. (#9565)"

This reverts commit 729cb5eec6.

* Revert "Dont insert shred payload into rocksdb (#9366)"

This reverts commit 5ed39de8c5.
2020-04-24 15:04:23 -07:00
Greg Fitzgerald 76b1c2baf0
One less alloc per transaction (#9705)
* One less alloc per transaction

* Fix benches

* Fix compiler warnings in bench build

* Fix move build

* Fix bench
2020-04-24 13:03:46 -06:00
dependabot-preview[bot] d2cef8ed9b
Bump num_cpus from 1.12.0 to 1.13.0 (#9682)
* Bump num_cpus from 1.12.0 to 1.13.0

Bumps [num_cpus](https://github.com/seanmonstar/num_cpus) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases)
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.12.0...v1.13.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-04-23 13:08:06 -07:00
sakridge 504160b11f
Update to rocksdb 0.14 and set max wal size (#9668) 2020-04-23 08:38:09 -07:00
sakridge ecb343c23b
reduce errors (#9669)
Co-authored-by: Anatoly Yakovenko <anatoly@solana.com>
2020-04-22 19:46:06 -07:00
Tyera Eulberg 5d2f488004
Add getLowestNonpurgedBlock rpc; use blockstore api in getConfirmedBlocks (#9656)
automerge
2020-04-22 13:33:06 -07:00
dependabot-preview[bot] b1ca74ed30 Bump regex from 1.3.6 to 1.3.7
Bumps [regex](https://github.com/rust-lang/regex) from 1.3.6 to 1.3.7.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.3.6...regex-1.3.7)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-04-20 08:02:22 -07:00
sakridge a5419fe79e
Error for invalid shred. (#9588) 2020-04-19 21:15:09 -07:00
anatoly yakovenko 729cb5eec6
Untar is called for shred archives that do not exist. (#9565)
automerge
2020-04-18 08:34:55 -07:00
Michael Vines 47ae57610a Only build x86_64-unknown-linux-gnu on docs.rs 2020-04-16 19:06:17 -07:00
anatoly yakovenko 5ed39de8c5
Dont insert shred payload into rocksdb (#9366)
automerge
2020-04-16 18:20:55 -07:00
sakridge 66abe45ea1
Decouple accounts hash calculation from snapshot hash (#9507) 2020-04-16 15:12:20 -07:00
Stephen Akridge 30b3862770 Don't unwrap on session new 2020-04-16 08:25:45 -07:00
Tyera Eulberg 530c542002
Rpc: Speed up getBlockTime (#9510)
* Add get-block-time metrics

* Add datapoints to blockstore rpc apis

* Tune timestamp_slot_range

* Refactor get_timestamp_slots

* Cargo.lock
2020-04-15 18:09:14 -06:00
Ryo Onodera c77ed82caa
Use same max_age regardless of leader/not-leader (#9423)
automerge
2020-04-14 00:34:41 -07:00
Ryo Onodera 2d0224b64e
Switch reed-solomon-erasure from vendored to upstream (#9483) 2020-04-14 14:33:21 +09:00
dependabot-preview[bot] a0f0e199b7 Bump libc from 0.2.68 to 0.2.69
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.68 to 0.2.69.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.68...0.2.69)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-04-13 17:30:02 -07:00
Tyera Eulberg 91159ea8e3
Rpc: Add getConfirmedSignaturesForAddress (#9407)
automerge
2020-04-09 20:21:31 -07:00
Michael Vines d5a9ee97f2
Add --allow-dead-slots argument to `slot`/`print`/`json` commands (#9408)
automerge
2020-04-09 20:10:51 -07:00
Michael Vines f3afe5c99c
Remove dead code (#9404)
automerge
2020-04-09 13:09:59 -07:00
Tyera Eulberg e1aa247548
Rpc: Add getConfirmedTransaction (#9381)
* Add blockstore method to return a complete transaction by signature

* Plumb getConfirmedTransaction rpc

* Add doc
2020-04-09 00:57:30 -06:00
Tyera Eulberg f655b3f0fd
Fix lowest_cleanup_slot check in Blockstore (#9382)
automerge
2020-04-08 18:47:16 -07:00
carllin 4522e85ac4
Add Metrics/Dashboards tracking block production (#9342)
* Add metric tracking blocks/dropped blocks

Co-authored-by: Carl <carl@solana.com>
2020-04-08 14:35:24 -07:00
Tyera Eulberg 36e73cada4
Add blockstore address-to-signature index (#9367)
automerge
2020-04-08 12:50:39 -07:00
Tyera Eulberg acf64f8476
Update getSignatureStatuses to return historical statuses (#9314)
automerge
2020-04-06 03:04:54 -07:00
Tyera Eulberg 49e2cc6593
Rework TransactionStatus index in blockstore (#9281)
automerge
2020-04-04 20:24:06 -07:00
Michael Vines ad0997e15f
RPC: add `err` field to TransactionStatus, alongside the now deprecated `status` field (#9296)
automerge
2020-04-04 16:13:26 -07:00
dependabot-preview[bot] 8cdf406dd3 Bump serde from 1.0.105 to 1.0.106
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.105 to 1.0.106.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.105...v1.0.106)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-04-04 15:13:52 -07:00
Michael Vines b557b3170e Add log before opening database 2020-04-03 14:55:06 -07:00
sakridge 9493de4443
Add snapshot compression option (#9276) 2020-04-03 13:13:49 -07:00
Jack May d61191db40
fix bench warnings (#9277) 2020-04-02 21:56:38 -07:00
sakridge 73e99cc513
Ledger cleanup fixes (#9131)
* Fix purging happening every slot when cleanup service is not started at slot 0
* Purge by shred count instead of slots since slots can have variable
number of shreds
2020-03-30 19:02:12 -07:00
Michael Vines d7fa40087c
Bump version to 1.2.0 2020-03-28 09:44:13 -07:00
sakridge 2863f8ec65
Use 1gb as genesis limit to fix bench-tps ledger from not starting (#9133)
automerge
2020-03-27 16:50:19 -07:00
Michael Vines b130c298df
Remove chatty 'setting snapshot root:' info log (#9122) 2020-03-27 10:24:59 -07:00
sakridge b7b4aa5d4d
move rpc types from client to client-types crate (#9039)
* Separate client types into own crate, so ledger does not need it

Removes about 50 crates of dependency from ledger

* Drop Rpc name from transaction-status types
2020-03-26 13:29:30 -07:00
sakridge ed036b978d
Accumulate blockstore metrics and submit every 2s (#9075) 2020-03-26 12:51:41 -07:00
Michael Vines b8fda9d730
Log how much data the ledger holds before processing it (#9079) 2020-03-25 21:41:50 -07:00
dependabot-preview[bot] 095c79e863
Bump regex from 1.3.5 to 1.3.6 (#9055)
Bumps [regex](https://github.com/rust-lang/regex) from 1.3.5 to 1.3.6.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.3.5...1.3.6)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-25 17:24:14 -07:00
carllin 9dc69d9843
Store and compute node/stake state in EpochStakes struct (#8958)
* Store and compute needed bank state in EpochStakes struct
2020-03-25 12:19:15 -07:00
Ryo Onodera f987c18a7e
Strictly validate the contents of snapshot/genesis (#8959)
automerge
2020-03-25 02:46:41 -07:00
Michael Vines 70c167182a
ledger tool now outputs transaction status information if available (#9024)
automerge
2020-03-23 12:49:21 -07:00
sakridge d75a470ffa
Ledger processing speed tracking (#9005) 2020-03-23 12:19:11 -07:00
Tyera Eulberg 1b8f9e75dd
Update getSignatureStatus: support multiple signatures, include slot in each response item (#9022)
* Rename enable-rpc-get-confirmed-block

* Rename RpcTransactionStatus -> RpcTransactionStatusMeta

* Return simplified RpcTransactionStatus; Add support for multiple transactions

* Update docs

* typo
2020-03-23 11:25:39 -06:00
sakridge 4d2b83d01f
Add option to disable rocks compaction (#9011) 2020-03-23 08:42:32 -07:00
Michael Vines 88ba8439fc
Add frozen account support (#8989)
automerge
2020-03-22 11:10:04 -07:00
sakridge 4dd0367136
Rwlock storage opt (#9006)
* Remove unecessary account paths rwlock

* Remove path rwlock in accounts_db and optimize storage critical section
2020-03-22 10:04:03 -07:00
carllin dc1db33ec9
Add Capabilities to Signal BroadcastStage to Retransmit (#8899) 2020-03-19 23:35:01 -07:00
Michael Vines f78a90bce2
Vote InitializeAccount and UpdateNode instructions now need a signature from the validator identity (#8947)
automerge
2020-03-19 01:58:52 -07:00
dependabot-preview[bot] 24d871b529
Bump serde from 1.0.104 to 1.0.105 (#8954)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.104 to 1.0.105.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.104...v1.0.105)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-18 22:35:30 -07:00
dependabot-preview[bot] 8f5ee6832f
Bump libc from 0.2.67 to 0.2.68 (#8915)
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.67 to 0.2.68.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.67...0.2.68)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-18 13:27:08 -06:00
Michael Vines bf60345b7a
Remove all snapshots not matching the desired hash 2020-03-17 22:58:01 -07:00
sakridge 7079559c2d
Fix windows build by removing sys-info (#8860)
Doesn't build for windows.
2020-03-16 12:53:13 -07:00
Carl 009c124fac Remove generic 2020-03-16 07:57:07 -07:00
Carl 7029c88305 use matches macro 2020-03-16 07:57:07 -07:00
Carl 9411fc00b8 Lower error level 2020-03-16 07:57:07 -07:00
Michael Vines b7b36bb0a4
Upgrade to Rust 1.42 (#8836)
* Upgrade to Rust 1.42

* deref

* parens

Co-authored-by: Trent Nelson <trent@solana.com>
2020-03-13 14:15:22 -06:00
dependabot-preview[bot] 52b254071c
Bump regex from 1.3.4 to 1.3.5 (#8830)
Bumps [regex](https://github.com/rust-lang/regex) from 1.3.4 to 1.3.5.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.3.4...1.3.5)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-13 07:43:07 -06:00
dependabot-preview[bot] f93c8290f4
Bump sys-info from 0.5.9 to 0.5.10 (#8810)
Bumps [sys-info](https://github.com/FillZpp/sys-info-rs) from 0.5.9 to 0.5.10.
- [Release notes](https://github.com/FillZpp/sys-info-rs/releases)
- [Changelog](https://github.com/FillZpp/sys-info-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/FillZpp/sys-info-rs/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-12 13:42:31 -06:00
Jack May 6eb4973780
Don't use move semantics if not needed (#8793) 2020-03-11 14:37:23 -07:00
dependabot-preview[bot] e41ff2df66
Bump chrono from 0.4.10 to 0.4.11 (#8755)
Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.10 to 0.4.11.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/master/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.10...v0.4.11)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-10 08:43:18 -06:00
dependabot-preview[bot] f88b79d42b
Bump itertools from 0.8.2 to 0.9.0 (#8756)
Bumps [itertools](https://github.com/bluss/rust-itertools) from 0.8.2 to 0.9.0.
- [Release notes](https://github.com/bluss/rust-itertools/releases)
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/bluss/rust-itertools/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-10 08:42:11 -06:00
dependabot-preview[bot] 1a0dd53450
Bump rayon from 1.2.0 to 1.3.0 (#8757)
Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.2.0 to 1.3.0.
- [Release notes](https://github.com/rayon-rs/rayon/releases)
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md)
- [Commits](https://github.com/rayon-rs/rayon/compare/v1.2.0...rayon-core-v1.3.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-10 08:40:02 -06:00
carllin 9872430bd2
Add VoteTracker for tracking cluster's votes in gossip (#8327)
Track votes by slot in cluster_vote_listener
2020-03-09 22:03:09 -07:00
dependabot-preview[bot] ff74452ef3
Bump libc from 0.2.66 to 0.2.67 (#8680)
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.66 to 0.2.67.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.66...0.2.67)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-08 21:38:58 -07:00
Greg Fitzgerald 2242b1b4a5
Bump byteorder from 1.3.2 to 1.3.4 (#8159)
Bumps [byteorder](https://github.com/BurntSushi/byteorder) from 1.3.2 to 1.3.4.
- [Release notes](https://github.com/BurntSushi/byteorder/releases)
- [Changelog](https://github.com/BurntSushi/byteorder/blob/master/CHANGELOG.md)
- [Commits](https://github.com/BurntSushi/byteorder/compare/1.3.2...1.3.4)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-03-05 17:07:18 -07:00
carllin f47a789b15
Add find_incomplete_slots (#8654)
* Add find_incomplete_slots

* Add live slots iterator
2020-03-05 10:58:00 -08:00
Grimes 01607b9860
Add NextSlotsIterator (#8652)
automerge
2020-03-04 20:46:58 -08:00
sakridge 23d8c7ff0e
Generate a snapshot at synchronized points (#8532)
Co-authored-by: anatoly yakovenko <anatoly@solana.com>
2020-03-04 19:23:40 -08:00
carllin dec3da8f9d
Add orphan iterator (#8636) 2020-03-04 18:10:30 -08:00
sakridge b68b74ac32
Check transaction signatures in entry verify (#8596) 2020-03-03 20:49:51 -08:00
sakridge b084c1d437
Remove accounts hack and correctly restore accounts store counts (#8569)
* Remove accounts hack and correctly restore append-vec counts

* Add test
2020-03-03 20:48:55 -08:00
Ryo Onodera d86103383a
Do periodic inbound cleaning for rooted slots (#8436)
* Do periodic inbound compaction for rooted slots

* Add comment

* nits

* Consider not_compacted_roots in cleanup_dead_slot

* Renames in AccountsIndex

* Rename to reflect expansion of removed accounts

* Fix a comment

* rename

* Parallelize clean over AccountsIndex

* Some niceties

* Reduce locks and real chunked parallelism

* Measure each step for sampling opportunities

* Just noticed par iter is maybe lazy

* Replace storage scan with optimized index scan

* Various clean-ups

* Clear uncleared_roots even if no updates
2020-03-03 14:57:25 +09:00
Ryo Onodera 5d9130a3c4 Hack to skip cleanup_dead_slots upon snapshot load 2020-03-02 10:24:12 -07:00