Commit Graph

841 Commits

Author SHA1 Message Date
sakridge edf2b1ee85
Check for deleting key, make sure list is empty again (#11007) 2020-07-11 20:38:34 -07:00
Greg Fitzgerald 145906123a
Simplify use of SendTransactionService (#10999)
* Send transaction upon recv

This will allow us to move the channel to the public interface

* Use a channel, not a method, to communicate

* Pipeline the services

* Ignore unused return values

* Fix clippy warning
2020-07-11 01:14:41 +00:00
Jack May 57576b07ef
Fix warnings (#10992)
* Fix warnings

* disable warning
2020-07-10 20:02:55 +00:00
sakridge 631f051c68
Add jitter to cleanup to prevent all nodes cleaning at the same time (#10936) 2020-07-10 11:54:45 -07:00
sakridge 1f5070e569
Add custom print for Bloom object (#10971) 2020-07-10 11:53:24 -07:00
Ryo Onodera f1c1152948
Fix skipped slot detection for eager rent collect (#10890) 2020-07-10 08:49:18 +09:00
Tyera Eulberg 25228ca957
Fix nonce fee_calculator overwrite (#10973)
* Add failing test

* Pass fee_calculator to prepare_if_nonce_account; only overwrite in error case
2020-07-09 20:22:21 +00:00
Greg Fitzgerald 16eeea4f82
Move SendTransactionService to solana_runtime (#10972) 2020-07-09 18:28:26 +00:00
Greg Fitzgerald ebadbce920
Drop bank from BlockCommitmentCache (#10959)
* Remove bank reference from BlockCommitmentCache

* Don't use a Bank to create BlockCommitmentCache

* Rename recent_slot to slot
2020-07-09 00:50:13 +00:00
dependabot-preview[bot] 841ecfd927
chore(deps): bump bincode from 1.2.1 to 1.3.1 (#10867)
* chore(deps): bump bincode from 1.2.1 to 1.3.1

Bumps [bincode](https://github.com/servo/bincode) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/servo/bincode/releases)
- [Commits](https://github.com/servo/bincode/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

* Switch from deprecated method

* Add options to maintain behavior with bincode::options()

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-07-09 00:08:05 +00:00
sakridge 6429042b6e
Add --print-account-stats/--snapshot-archive-path arguments to ledger-tool (#10945)
Allows for seeing how the accounts are being stored
and specifying a different snapshot source directory.
2020-07-08 09:32:11 -07:00
Greg Fitzgerald 2fdbb97244
Rename largest_confirmed_root to highest_confirmed_root (#10947) 2020-07-07 23:59:46 +00:00
sakridge 1269e348fb
Add a way to re-hash the bank (#10765) 2020-07-06 12:42:41 -07:00
Ryo Onodera 39b3ac6a8d
Introduce automatic ABI maintenance mechanism (2/2; rollout) (#8012)
* Introduce automatic ABI maintenance mechanism (2/2; rollout)

* Fix stable clippy

* Change to symlink

* Freeze abi of Tower

* fmt...

* Improve dev-experience!

* Update BankSlotDelta

$ diff -u /tmp/abi8/*7dg6BreYxTuxiVz6aLvk3p2Z7GQk2cJqfGvC9h4FAoSj* /tmp/abi8/*9chBcbXVJ4fK7uGgydQzam5aHipaAKFw6V4LDFpjbE4w*
--- /tmp/abi8/bank__BankSlotDelta_frozen_abi__test_abi_digest_7dg6BreYxTuxiVz6aLvk3p2Z7GQk2cJqfGvC9h4FAoSj      2020-06-18 18:01:22.831228087 +0900
+++ /tmp/abi8/bank__BankSlotDelta_frozen_abi__test_abi_digest_9chBcbXVJ4fK7uGgydQzam5aHipaAKFw6V4LDFpjbE4w      2020-07-03 15:59:58.430695244 +0900
@@ -140,7 +140,7 @@
                                                         field u8
                                                             primitive u8
                                                         field solana_sdk::instruction::InstructionError
-                                                            enum InstructionError (variants = 34)
+                                                            enum InstructionError (variants = 35)
                                                                 variant(0) GenericError (unit)
                                                                 variant(1) InvalidArgument (unit)
                                                                 variant(2) InvalidInstructionData (unit)
@@ -176,6 +176,7 @@
                                                                 variant(31) CallDepth (unit)
                                                                 variant(32) MissingAccount (unit)
                                                                 variant(33) ReentrancyNotAllowed (unit)
+                                                                variant(34) MaxSeedLengthExceeded (unit)
                                                     variant(9) CallChainTooDeep (unit)
                                                     variant(10) MissingSignatureForFee (unit)
                                                     variant(11) InvalidAccountIndex (unit)

* Fix some merge conflicts...
2020-07-06 20:22:23 +09:00
sakridge 832d47317e
Move clean accounts to background service (#10898) 2020-07-02 22:25:17 -07:00
carllin 7a71580d53
Add heaviest subtree utility functions (#10863)
Co-authored-by: Carl <carl@solana.com>
2020-07-01 08:45:32 +00:00
Ryo Onodera b89e506cbb
Consistently create temp dirs under ledger/farf (#10848) 2020-07-01 11:38:59 +09:00
Ryo Onodera cabedc0f36
Remove outdated todo comment (#10850) 2020-06-30 13:43:01 +00:00
Greg Fitzgerald 50b3fa83a0
Move BankCommitmentCache to solana_runtime (#10816)
* Remove Blockstore member variable from BlockCommitmentCache

* Hoist is_confirmed_rooted() to its only caller

BlockCommitmentCache no longer depends on Blockstore

* Move BlockCommitmentCache to solana_runtime
2020-06-25 22:06:58 -06:00
sakridge 7430978d1a
Keep oldest snapshot so playing the ledger is possible from local data. (#10814) 2020-06-25 20:41:44 -07:00
Ryo Onodera 50f7ed80c8
Dont skip eager rent collect across gapped epochs (#10206)
* Dont skip eager rent collect across gapped epochs

* Adjust style and comment

* Adjust ascii chart and comment a bit

* Moar assert

* Relax the partition_count assert for completeness

* Tweak comment...

* tweak a bit

* Add gating logic

* Address reviews

* small formatting

* Clarify the code by replacing auto_generated...

* small formatting

* small formatting

* small formatting

* small formatting

* Narrow down conditional compilation scope
2020-06-25 23:11:33 +09:00
Greg Fitzgerald 7ade330b23
Rename Client methods to match proposed BanksClient (#10793) 2020-06-25 03:35:38 +00:00
Greg Fitzgerald 1c498369b5
Remove fee-payer guesswork from Message and Transaction (#10776)
* Make Message::new_with_payer the default constructor

* Remove Transaction::new_[un]signed_instructions

These guess the fee-payer instead of stating it explicitly
2020-06-24 14:52:38 -06:00
Kristofer Peterson 6d81eede93
Add CLI options and runtime support for selection of output snapshot version. (#10536) 2020-06-19 14:38:37 +09:00
Greg Fitzgerald 0550b893b0
Fix typos (#10675)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-06-17 20:54:52 -07:00
Greg Fitzgerald 26ebfedf46
Derive the working bank (#10670) 2020-06-17 20:16:57 -06:00
Jack May dac7dc2f10
Move types to more appropriate files (#10638) 2020-06-17 10:39:14 -07:00
Greg Fitzgerald 6ee222363e
Move BankForks to solana_runtime (#10637)
* Move BankForks to solana_runtime

* Update imports
2020-06-17 15:27:03 +00:00
Kristofer Peterson 50c93d4441
Cleanup error type mapping in serde_snapshot (#10580)
* Cleanup error type mapping in serde_snapshot

* Fail when account storage entry files cannot be moved from snapshot
2020-06-17 17:56:29 +09:00
carllin 8bd62d78eb
Fix race in remove_unrooted_slot (#10607)
* Fix race

* clippy fixes

* Rename and add comment

Co-authored-by: Carl <carl@solana.com>
2020-06-15 19:41:26 -07:00
Jack May b6a9573748
Route all loader messages to log collector (#10528) 2020-06-13 13:20:08 -07:00
Ryo Onodera dfe72d5242
Optimize stale slot shrinking for previously cleaned roots (#10099)
* Prioritize shrinking of previously cleaned roots

* measure time of stale slot shrinking

* Disable shrink for test

* shrink: budgeting, store count, force for snapshot

* Polish implementation and fix tests

* Fix ci..

* Clean up a bit

* Further polish implementation and fix/add tests

* Rebase fixes

* Remove unneeded Default for AccountStorageEntry

* Address review comments

* More cleanup

* More cleanup
2020-06-12 14:51:43 +09:00
Jack May 4f761395d2
Adhere to naming conventions (#10530)
automerge
2020-06-11 19:10:34 -07:00
carllin 2e1d59ff85
Adopt heaviest subtree fork choice rule (#10441)
* Add HeaviestSubtreeForkChoice

* Make replay stage switch between two fork choice rules

Co-authored-by: Carl <carl@solana.com>
2020-06-11 12:16:04 -07:00
Boqin Qin 1e3554b33d
runtime: fix possible deadlock in bank (#10466) 2020-06-11 02:02:08 +09:00
Boqin Qin 0a638a8f17
runtime: fix possible deadlock in accounts_db (#10469) 2020-06-10 19:37:15 +09:00
Greg Fitzgerald 4131eee94d
Expose last-valid-slot to BankClient and ThinClient users (#10478)
automerge
2020-06-09 17:07:32 -07:00
Trent Nelson b329e6d1a4 Add --warp-slot argument to |solana-ledger-tool create-snapshot| 2020-06-09 09:16:21 -07:00
Kristofer Peterson e23340d89e
Clippy cleanup for all targets and nighly rust (also support 1.44.0) (#10445)
* address warnings from 'rustup run beta cargo clippy --workspace'

minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs

expect some performance improvement AccountsDB::clean_accounts()

* address warnings from 'rustup run beta cargo clippy --workspace --tests'

* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'

* rustfmt

* fix warning stragglers

* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned

* Remove lock around JsonRpcRequestProcessor (#10417)

automerge

* make ancestors parameter optional to avoid forcing construction of empty hash maps

Co-authored-by: Greg Fitzgerald <greg@solana.com>
2020-06-09 09:38:14 +09:00
Michael Vines 136342eaae Bump new_system_program_activation_epoch by 2 2020-06-08 09:39:42 -07:00
Michael Vines 7e2651ca51
RPC simulateTransaction endpoint now returns program log output (#10432) 2020-06-06 10:18:28 -07:00
Justin Starry 754f25ae99
Avoid AccountInUse errors when simulating transactions (#10391)
automerge
2020-06-04 19:06:01 -07:00
Michael Vines 26e684f375
Deactivate legacy_system_instruction_processor at epoch 58/38 (preview/stable) (#10406)
automerge
2020-06-04 00:01:47 -07:00
Jack May 61d91d2d55
Don't reuse executable accounts between instructions (#10403)
automerge
2020-06-03 21:16:15 -07:00
Michael Vines 21d62493ff Add legacy_system_instruction_processor0 tests 2020-06-03 16:04:06 -07:00
Michael Vines 9c631a01c1 Enable rolling upgrade of system_instruction_processor 2020-06-03 16:04:06 -07:00
Michael Vines 1331c9a680 Add legacy_system_instruction_processor 2020-06-03 16:04:06 -07:00
Michael Vines a4cd96609c
Add built-in programs to InvokeContext (#10383)
automerge
2020-06-03 12:48:19 -07:00
Greg Fitzgerald 5d248fe5f8
Permit paying oneself (#10337)
* Allow paying to oneself

* cargo fmt

* Permit pay-to-self in CLI

No test here because we're just removing an [untested] special case.

Fixes #10339
2020-05-30 20:12:48 -06:00
Tyera Eulberg b563b49ed5
Improve Rpc inflation tooling (#10309)
automerge
2020-05-29 11:50:25 -07:00
Michael Vines e3b444834f
Add more logging while unpacking snapshots (#10266) 2020-05-27 10:41:05 -07:00
Tyera Eulberg 4e431bc818
Add mechanism to get blockhash's last valid slot (#10239)
automerge
2020-05-26 12:06:21 -07:00
Jack May 03abd3ddd7
Prevent privilege escalation (#10232)
automerge
2020-05-26 01:02:31 -07:00
Greg Fitzgerald 36a36d1c83
No longer allow create-account to add funds to an existing account (#10192)
automerge
2020-05-22 16:39:01 -07:00
Kristofer Peterson b7a32f01c0
Multi-version snapshot support (#9980)
* Multi-version snapshot support

* rustfmt

* Remove CLI options and runtime support for selection output snapshot version.
Address some clippy complaints.

* Muzzle clippy type complexity warning.

Despite clippy's suggestion, it is not currently possible to create type aliases
for traits and so everything within the 'Box<...>' cannot be type aliased.

This then leaves creating full blown traits, and either implementing
said traits by closure (somehow) or moving the closures into new structs
implementing said traits which seems a bit of a palaver.

Alternatively it is possible to define and use the type alias 'type ResultBox<T> = Result<Box<T>>'
which does seems rather pointless and not a great reduction in complexity but is enough to keep clippy happy.

In the end I simply went with squelching the clippy warning.

* Remove now unused Serialize/Deserialize trait implementations for AccountStorageEntry and AppendVec

* refactor versioned de/serialisers

* rename serde_utils to serde_snapshot

* move call to accounts_db.generate_index() back down to context_accountsdb_from_stream()

* update version 1.1.1 to 1.2.0
remove nested use of serialize_bytes

* cleanups

* Add back measurement of account storage entry serialization.
Remove construction of Vec and HashMap temporaries during serialization.

* consolidate serialisation test cases into serde_snapshot.
clean up leakage of implementation details in serde_snapshot.

* move short term / legacy snapshot code into child module

* add serialize_iter_as_tuple

* preliminary integration of following commit

commit 6d58b73c47294bfb93465d5a83cd2175660b6e6d
Author: Ryo Onodera <ryoqun@gmail.com>
Date:   Wed May 20 14:02:02 2020 +0900

    Confine snapshot 1.1 relic to versioned codepath

* refactored serde_snapshot, rustfmt
legacy accounts_db format now "owns" both leading u64s, legacy bank_rc format has none

* reduce type complexity (clippy)
2020-05-23 02:54:24 +09:00
sakridge ce17de7d25
Add option to wait for a specific epoch length to bench-tps (#10083) 2020-05-20 16:42:46 -07:00
Jack May 4a72c2b054
Support cross-program invocation to native programs (#10136) 2020-05-20 09:24:57 -07:00
Tyera Eulberg 9d89fb5c35
Fix another unstable test after eager rent (#10120) 2020-05-20 09:57:33 -06:00
Jack May 36627fb8b3
move builtin programs out of bank (#10132)
automerge
2020-05-19 19:45:30 -07:00
Jack May 431a228402
fix clock bankhash mismatch (#10078)
automerge
2020-05-18 19:48:06 -07:00
Michael Vines bbc549f592
Rename program_id to owner in system instructions (#10069) 2020-05-18 12:55:41 -07:00
Kristofer Peterson 4ca352a344
Use serde provided serialization for atomics (#10096)
automerge
2020-05-18 08:30:27 -07:00
Greg Fitzgerald 5e89bd8868
Panic if no fee-payer found via Message::new() (#10050)
automerge
2020-05-15 12:23:09 -07:00
Kristofer Peterson 58ef02f02b
9951 clippy errors in the test suite (#10030)
automerge
2020-05-15 09:35:43 -07:00
Jack May eb1acaf927
Remove archiver and storage program (#9992)
automerge
2020-05-14 18:22:47 -07:00
Ryo Onodera 13bc3f8094
Fix unstable test after eager rent collection (#10031)
automerge
2020-05-13 10:35:58 -07:00
Ryo Onodera 1eb40c3fe0
Introduce eager rent collection (#9527)
* Switch AccountsIndex.account_maps from HashMap to BTreeMap

* Introduce eager rent collection

* Start to add tests

* Avoid too short eager rent collection cycles

* Add more tests

* Add more tests...

* Refacotr!!!!!!

* Refactoring follow up

* More tiny cleanups

* Don't rewrite 0-lamport accounts to be deterministic

* Refactor a bit

* Do hard fork, restore tests, and perf. mitigation

* Fix build...

* Refactor and add switch over for testnet (TdS)

* Use to_be_bytes

* cleanup

* More tiny cleanup

* Rebase cleanup

* Set Bank::genesis_hash when resuming from snapshot

* Reorder fns and clean ups

* Better naming and commenting

* Yet more naming clarifications

* Make prefix width strictly uniform for 2-base partition_count

* Fix typo...

* Revert cluster-dependent gate

* kick ci?

* kick ci?

* kick ci?
2020-05-13 16:22:14 +09:00
Justin Starry 5cc252d471
Remove hash field from account (#9915) 2020-05-12 23:39:46 +08:00
Jack May 97e17f9b32 Programs can only sign their accounts 2020-05-11 09:06:05 -07:00
Tyera Eulberg 3ee702a922
Rpc: Add getCirculatingSupply endpoint, redux (#9953)
* Add Bank.clock() helper

* Add non-circulating calculations

* Plumb getSupply rpc endpoint

* Add docs for getSupply, and remove getTotalSupply from docs

* Add pubkeys! procedural macro

* Use procedural macro in non_circulating_supply
2020-05-09 12:05:29 -06:00
Ryo Onodera 00e45ec935
Maintain sysvar balances for consistent market cap. (#9936)
* Maintain sysvar balances for consistent market cap.

* Unindent
2020-05-09 02:42:32 +09:00
Ryo Onodera 2ac50177a6
Include account.owner into account hash (#9917)
automerge
2020-05-07 13:01:11 -07:00
Tyera Eulberg 754c65c066
Refactor RPC subscriptions account handling (#9888)
* Switch subscriptions to use commitment instead of confirmations

* Add bank method to return account and last-modified slot

* Add last_modified_slot to subscription data and use to filter account subscriptions

* Update tests to non-zero last_notified_slot

* Add accounts subscriptions to test; fails at higher tx load

* Pass BankForks to RpcSubscriptions

* Use BankForks on add_account_subscription to properly initialize last_notified_slot

* Bundle subscriptions

* Check for non-equality

* Use commitment to initialize last_notified_slot; revert context.slot chage
2020-05-07 00:23:06 -06:00
carllin 3442f36f8a
Repair alternate versions of dead slots (#9805)
Co-authored-by: Carl <carl@solana.com>
2020-05-05 14:07:21 -07:00
Tyera Eulberg f5b0d13f08
Rpc: add getLargestAccounts endpoint (#9869)
automerge
2020-05-04 16:46:10 -07:00
sakridge f37f83fd12
Fuzzer test and fixes (#9853) 2020-05-02 08:07:52 -07:00
sakridge 894549f002
Put empty accounts in the accounts list on load (#9840)
Indexing into accounts array does not match account_keys otherwise.
Also enforce program accounts not at index 0
Enforce at least 1 Read-write signing fee-payer account.
2020-05-01 17:23:33 -07:00
Michael Vines 8dfe0affd4
Add incinerator sysvar (#9815) 2020-04-30 22:04:08 -07:00
Michael Vines 230df0ec0c
Upgrade to Rust 1.43.0 (#9754) 2020-04-29 18:02:05 -07:00
Jack May 068f12fd6f
Add Cross-program invocations (#9582) 2020-04-28 14:33:56 -07:00
sakridge aac580686f
fix test compilation from add_static_program change (#9755) 2020-04-27 23:37:08 -07:00
Jack May efad193180
Make default programs static (#9717) 2020-04-27 21:05:12 -07:00
anatoly yakovenko 8ef097bf6f
Input values are not sanitized after they are deserialized, making it far too easy for Leo to earn SOL (#9706)
* sanitize gossip protocol messages
* sanitize transactions
* crds protocol sanitize
2020-04-27 11:06:00 -07:00
Ryo Onodera 9918539229
Introduce type alias Ancestors (#9699)
* Introduce type alias AncestorList

* Rename AncestorList => Ancestors
2020-04-27 11:07:03 +09:00
sakridge e907c0e650
Filter program ids to store (#9721)
automerge
2020-04-26 00:11:37 -07:00
Greg Fitzgerald 76b1c2baf0
One less alloc per transaction (#9705)
* One less alloc per transaction

* Fix benches

* Fix compiler warnings in bench build

* Fix move build

* Fix bench
2020-04-24 13:03:46 -06:00
Jack May b1a0abc7a6
Bump libloading to v0.6.1 (#9615)
automerge
2020-04-20 16:46:06 -07:00
Jack May d08517db8c
Nit picks (#9580) 2020-04-18 22:39:08 -07:00
Jack May 58887c591b
Add and update tests (#9566) 2020-04-18 17:04:13 -07:00
Jack May 241a05fc52
Add native loader entry points (#9486) 2020-04-15 09:41:29 -07:00
sakridge 217828a849
Reduce accounts_db prints (#9469) 2020-04-15 09:05:54 -07:00
Ryo Onodera c77ed82caa
Use same max_age regardless of leader/not-leader (#9423)
automerge
2020-04-14 00:34:41 -07:00
sakridge 99655206c8
Calculate account refs fix (#9447) 2020-04-11 12:52:10 -07:00
Jack May ad0482be73
Revert "Add native loader entry points (#9275)" Breaks genesis_config abi (#9377)
This reverts commit ed86d8d1fc.
2020-04-08 14:36:18 -07:00
carllin 4522e85ac4
Add Metrics/Dashboards tracking block production (#9342)
* Add metric tracking blocks/dropped blocks

Co-authored-by: Carl <carl@solana.com>
2020-04-08 14:35:24 -07:00
Ryo Onodera b28ec430e4
Introduce background stale AppendVec shrink mechanism (#9219)
* Introduce background AppendVec shrink mechanism

* Support ledger tool

* Clean up

* save

* save

* Fix CI

* More clean up

* Add tests

* Clean up yet more

* Use account.hash...

* Fix typo....

* Add comment

* Rename accounts_cleanup_service
2020-04-06 17:30:23 +09:00
Jack May ed86d8d1fc
Add native loader entry points (#9275) 2020-04-03 17:40:59 -07:00
Jack May 268e04cb4a
Rename CustomError to Custom (#9207) 2020-04-01 09:01:11 -07:00
Jack May 130c0b484d
Enforce an executable's rent exemption in the runtime (#9134) 2020-03-31 10:07:38 -07:00
Tyera Eulberg 50fa577af8
Use cluster confirmations in rpc and pubsub (#9138)
* Add runtime methods to simply get status and slot

* Add helper function to get slot confirmation_count from BlockCommitmentCache

* Return cluster confirmations in getSignatureStatus

* Remove use of invalid get_signature_confirmation_status

* Remove unused methods

* Update pubsub to use cluster confirmations

* Fix test_check_signature_subscribe failure

* Refactor confirmations to read commitment cache only once

* Review comments

* Use bank, root from BlockCommitmentCache

* Update docs

* Add metric for block-commitment aggregations

Co-authored-by: Justin Starry <justin@solana.com>
2020-03-30 17:53:25 -06:00