Commit Graph

315 Commits

Author SHA1 Message Date
Tyera Eulberg de5fb3ba0e
Blockstore address signatures: handle slots that cross primary indexes, and refactor get_confirmed_signatures_for_address2 (#11497)
* Freeze address-signature index in the middle of slot to show failure case

* Secondary filter on signature

* Use AddressSignatures iterator instead of manually decrementing slots

* Remove unused method

* Add metrics

* Add transaction-status-index doccumentation
2020-08-10 10:27:38 -06:00
carllin 7e25130529
Send votes from banking stage to vote listener (#11434)
*  Send votes from banking stage to vote listener

Co-authored-by: Carl <carl@solana.com>
2020-08-07 11:21:35 -07:00
Tyera Eulberg 5530ee4c95
Add address-based lower bound to get_confirmed_signatures_for_address2 loop (#11426) 2020-08-06 22:29:25 +00:00
Tyera Eulberg 1061b50665
Fix blockstore empty panic (#11423)
* Add panicking test

* Add failing test: fresh transaction-status column shouldn't point at valid root 0

* Prevent transaction status match outside of primary-index bounds

* Initialize transaction-status and address-signature primer entries with Slot::MAX

* Revert "Add failing test: fresh transaction-status column shouldn't point at valid root 0"

This reverts commit cbad2a9fae22e5531e3b4ff1b0a9d6a223826c71.

* Revert "Initialize transaction-status and address-signature primer entries with Slot::MAX"

This reverts commit ffaeac0669d0cbe18dd68b5ce177e15a92360b72.
2020-08-06 22:21:46 +00:00
Greg Fitzgerald 3fdd8ffdf0
Remove circular dep between InstructionError and SystemError (#11427) 2020-08-06 22:04:43 +00:00
Michael Vines 02c0981ecf Rename startAfter to before 2020-08-05 14:33:14 -07:00
Michael Vines a11f137810 Rework get_confirmed_signatures_for_address2 2020-08-05 14:33:14 -07:00
Michael Vines 1b2276520b Add getConfirmedSignaturesForAddress2 RPC method 2020-08-05 14:33:14 -07:00
Michael Vines 494968be66 UiTransactionEncoding is now copy 2020-08-05 14:27:12 -07:00
Michael Vines eefcf484cb clippy 2020-08-03 18:35:15 +00:00
carllin bf18524368
Add hook for getting vote transactions on replay (#11264)
* Add hook for getting vote transactions on replay

Co-authored-by: Carl <carl@solana.com>
2020-07-29 23:17:40 -07:00
Tyera Eulberg 6c38369042
Use OrderedIterator in TransactionStatusService (#11149)
* Split out get-first-err for unit testing

* Add failing test

* Add missing ordering
2020-07-21 20:06:49 +00:00
Tyera Eulberg 9a80e31bae
Expose tss to the other blockstore_processor path (#11070) 2020-07-14 21:14:48 -06:00
Michael Vines 491f5ae61a Add block time placeholder to getConfirmedBlock 2020-07-10 14:47:11 +00:00
dependabot-preview[bot] 841ecfd927
chore(deps): bump bincode from 1.2.1 to 1.3.1 (#10867)
* chore(deps): bump bincode from 1.2.1 to 1.3.1

Bumps [bincode](https://github.com/servo/bincode) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/servo/bincode/releases)
- [Commits](https://github.com/servo/bincode/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

* Switch from deprecated method

* Add options to maintain behavior with bincode::options()

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-07-09 00:08:05 +00:00
carllin 3f6042d8b3
Add RepairWeight to track votes seen in gossip for weighted repair (#10903)
* Add RepairWeight

Co-authored-by: Carl <carl@solana.com>
2020-07-06 22:49:40 -07:00
sakridge 58a475b789
Add db recovery methods (#10838) 2020-07-06 12:43:45 -07:00
sakridge 658de5b347
Skip and warn for hard-forks which are less than the start slot (#10918)
* Skip and warn for hard-forks which are less than the start slot

Option is used during a restart, but then after the restart is
complete, then the option is not needed if the starting slot
is past the hard-fork since the hard-fork should already be
in the snapshot it started from.

* Update ledger/src/blockstore_processor.rs

Co-authored-by: Michael Vines <mvines@gmail.com>

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-07-05 13:51:43 -07:00
carllin f014c94369
Add breakdown of erasure blobs (#10912)
Co-authored-by: Carl <carl@solana.com>
2020-07-03 17:44:32 -07:00
carllin f17ac70bb2
Add weighted traversal (#10877)
Co-authored-by: Carl <carl@solana.com>
2020-07-02 14:33:04 -07:00
Tyera Eulberg 2669ccb864
Update struct prefixes to Ui (#10874)
* Update comments

* Use Ui prefix
2020-07-01 20:06:40 +00:00
Ryo Onodera b89e506cbb
Consistently create temp dirs under ledger/farf (#10848) 2020-07-01 11:38:59 +09:00
sakridge 583cec922b
Remove ledger purge batching (#10830) 2020-06-29 14:44:35 -07:00
carllin 0fde0d7379
Add ancestor iterator to lib.rs (#10813)
Co-authored-by: Carl <carl@solana.com>
2020-06-25 23:39:55 +00:00
Greg Fitzgerald 1c498369b5
Remove fee-payer guesswork from Message and Transaction (#10776)
* Make Message::new_with_payer the default constructor

* Remove Transaction::new_[un]signed_instructions

These guess the fee-payer instead of stating it explicitly
2020-06-24 14:52:38 -06:00
sakridge 0e393a5684
Allow for hard fork at last root (#10762) 2020-06-23 21:49:11 -07:00
sakridge a1ef921b88
Rework backup and clear function (#10751) 2020-06-23 21:29:07 +00:00
carllin 77b8de193c
Add utility functions for testing (#10749)
* Add ancestor iterator

* Add blockstore generation from trees

Co-authored-by: Carl <carl@solana.com>
2020-06-23 12:05:00 -07:00
sakridge 2ba8fc5243
Remove slots past wait-for-supermajority slot. (#10720) 2020-06-22 20:27:25 -07:00
carllin a33fef9af2
Add repair breakdown by slot and index (#10717)
* Slot full logging

* Repair stats logging

Co-authored-by: Carl <carl@solana.com>
2020-06-19 18:28:15 -07:00
Greg Fitzgerald 0550b893b0
Fix typos (#10675)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-06-17 20:54:52 -07:00
Greg Fitzgerald 6ee222363e
Move BankForks to solana_runtime (#10637)
* Move BankForks to solana_runtime

* Update imports
2020-06-17 15:27:03 +00:00
Kristofer Peterson 50c93d4441
Cleanup error type mapping in serde_snapshot (#10580)
* Cleanup error type mapping in serde_snapshot

* Fail when account storage entry files cannot be moved from snapshot
2020-06-17 17:56:29 +09:00
sakridge 0de6c444d6
Simd poh (#10604)
* Simd poh

* Fix poh verify bench
2020-06-16 23:03:26 -07:00
sakridge 1eca9b19ab
Entry verify cleanup and gossip counters (#10632)
* Add prune message counter

* Switch to us verification time to match other counters

* Add separate transaction/poh verify timing
2020-06-16 14:00:29 -07:00
Greg Fitzgerald 2eb6f498a8
Remove redundant BankForks parameter (#10537) 2020-06-12 11:04:17 -06:00
Kristofer Peterson e23340d89e
Clippy cleanup for all targets and nighly rust (also support 1.44.0) (#10445)
* address warnings from 'rustup run beta cargo clippy --workspace'

minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs

expect some performance improvement AccountsDB::clean_accounts()

* address warnings from 'rustup run beta cargo clippy --workspace --tests'

* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'

* rustfmt

* fix warning stragglers

* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned

* Remove lock around JsonRpcRequestProcessor (#10417)

automerge

* make ancestors parameter optional to avoid forcing construction of empty hash maps

Co-authored-by: Greg Fitzgerald <greg@solana.com>
2020-06-09 09:38:14 +09:00
sakridge 2725acebea
Lower counter level (#10428) 2020-06-05 15:53:09 -07:00
Michael Vines aa6832964c
ledger_cleanup_service: compact at a slower rate than purging (#10414) 2020-06-04 21:06:06 -07:00
Ryo Onodera caa7f7a0c9
Support opening an in-use rocksdb as secondary (#10209)
automerge
2020-06-02 21:32:44 -07:00
Tyera Eulberg eee9a08376
Purge TransactionStatus and AddressSignatures exactly from ledger-tool (#10358)
* Add failing test

* Add execution path to purge primary-index columns exactly

* Fail gracefully if older TransactionStatus rocksdb keys are present

* Remove columns_empty check for special columns

* Move blockstore purge methods to submodule

* Remove unused column empty check
2020-06-02 19:49:31 -06:00
Greg Fitzgerald 19d11800bf
Remove WithSigner (#10325)
automerge
2020-05-29 23:17:44 -07:00
carllin bc37dfc4c1
log leader (#10316)
automerge
2020-05-29 04:35:20 -07:00
Kristofer Peterson fb4d8e1f62
cleanup clippy tests (#10172)
automerge
2020-05-29 00:26:06 -07:00
Michael Vines 5ac2ae1178
Purge next slots to avoid a blockstore_processor panic on restart (#10281) 2020-05-27 19:55:40 -07:00
Michael Vines e3b444834f
Add more logging while unpacking snapshots (#10266) 2020-05-27 10:41:05 -07:00
Michael Vines 7abdfc23f9
succceed is not a word 2020-05-26 23:17:50 -07:00
Michael Vines 156387aba4
LedgerCleanupService no longer causes an OOM and actually purges (#10199)
* cleanup_ledger() now services new_root_receiver while purging
* purge_slots() now fully deletes before compacting
* Add ledger pruning grafana graph
2020-05-24 21:41:54 -07:00
Kristofer Peterson b7a32f01c0
Multi-version snapshot support (#9980)
* Multi-version snapshot support

* rustfmt

* Remove CLI options and runtime support for selection output snapshot version.
Address some clippy complaints.

* Muzzle clippy type complexity warning.

Despite clippy's suggestion, it is not currently possible to create type aliases
for traits and so everything within the 'Box<...>' cannot be type aliased.

This then leaves creating full blown traits, and either implementing
said traits by closure (somehow) or moving the closures into new structs
implementing said traits which seems a bit of a palaver.

Alternatively it is possible to define and use the type alias 'type ResultBox<T> = Result<Box<T>>'
which does seems rather pointless and not a great reduction in complexity but is enough to keep clippy happy.

In the end I simply went with squelching the clippy warning.

* Remove now unused Serialize/Deserialize trait implementations for AccountStorageEntry and AppendVec

* refactor versioned de/serialisers

* rename serde_utils to serde_snapshot

* move call to accounts_db.generate_index() back down to context_accountsdb_from_stream()

* update version 1.1.1 to 1.2.0
remove nested use of serialize_bytes

* cleanups

* Add back measurement of account storage entry serialization.
Remove construction of Vec and HashMap temporaries during serialization.

* consolidate serialisation test cases into serde_snapshot.
clean up leakage of implementation details in serde_snapshot.

* move short term / legacy snapshot code into child module

* add serialize_iter_as_tuple

* preliminary integration of following commit

commit 6d58b73c47294bfb93465d5a83cd2175660b6e6d
Author: Ryo Onodera <ryoqun@gmail.com>
Date:   Wed May 20 14:02:02 2020 +0900

    Confine snapshot 1.1 relic to versioned codepath

* refactored serde_snapshot, rustfmt
legacy accounts_db format now "owns" both leading u64s, legacy bank_rc format has none

* reduce type complexity (clippy)
2020-05-23 02:54:24 +09:00
carllin 439fd30840
Fix erasure (#10095)
* Fix bad FEC blocks

* Add test

Co-authored-by: Carl <carl@solana.com>
2020-05-19 16:13:12 -07:00