Commit Graph

1658 Commits

Author SHA1 Message Date
steviez e69f37a464
Remove old function: account_balance_for_capitalization (#16383)
This function currently returns one of its' parameters and is thus useless.
2021-04-07 23:57:40 -05:00
Jack May 476fd40948
Add program-test tests to bpf programs (#16407) 2021-04-07 01:47:15 -07:00
Jack May 92f4018b07
Cleanup unsupported sysvars (#16390)
* Cleanup unsupported sysvars

* fix ser description
2021-04-06 00:08:03 -07:00
Jeff Washington (jwash) 95dc7b5449
reduce lock contention on latest_root (#16306) 2021-04-05 11:35:14 -05:00
Jeff Washington (jwash) 7a997759fa
reduce locking on get_rooted_entries (#16304) 2021-04-05 11:26:53 -05:00
Justin Starry 0596cf5405
Remove unprocessed transactions from log notifications (#16349) 2021-04-04 23:44:59 +08:00
Jeff Washington (jwash) 3996b699dc
read only account cache for executable accounts - improve replay (#16150)
* read only account cache

* tests

* clippy

* cleanup

* new file, add tests

* remove copy/paste code from test

* remove dead code

* all loads use cache

* remove stale comments

* add metrics logging for read only cache size

* report read only cache hits and misses

* consistency

* formatting

* rename, add comment

* u64

* better interaction with existing cache

* lru list saved between cleans
2021-04-01 07:16:34 -05:00
behzad nouri 3f63ed9a72
removes OrderedIterator and transaction batch iteration order (#16153)
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817

However load_and_execute_transactions is iterating over
  lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.

If iteratorion_order is [1, 2, 0], and i is 0, then:
  lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
  transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).

This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.
2021-03-31 23:59:19 +00:00
Jeff Washington (jwash) 416312b30b
update timings only on outer call (#16275)
* update timings only on outer call

* test & format
2021-03-31 18:02:59 -05:00
Jeff Washington (jwash) f374b35944
Hash stored accounts in bg (#16157)
* lazy calculate account hash

* push to bg thread

* remove deadlock

* logs

* format

* some cleanup on aisle 9

* format, fix up some metrics

* fix test, remove legacy function only there for tests

* cleanup

* remove unused store_hasher

* Switch to crossbeam

* clippy

* format

* use iter()

* rework from feedback

* hash_slot -> slot

* hash(cluster_type)

Co-authored-by: Carl Lin <carl@solana.com>
2021-03-31 15:39:34 -05:00
Jeff Washington (jwash) a5dcee254d
get rid of unnecessary data copy (#16214)
* get rid of unnecessary data copy

* preserve rent_epoch
2021-03-31 09:11:39 -05:00
sakridge 54c68ea83f
Drop write lock on sysvars (#15497)
* Drop write lock on sysvars

* adds env var for demoting sysvar write lock demotion

* moves demote logic to is_writable

* feature gates sysvar write lock demotion

* adds builtins to write lock demotion

* adds system program id to builtins

* adds Feature111...

* adds an abi-freeze test

* mvines set of builtin program keys

Co-authored-by: Michael Vines <mvines@gmail.com>

* update tests

* adds bpf loader keys

* Add test sysvar

* Plumb demote_sysvar to is_writable

* more plumbing of demote_sysvar_write_locks to is_writable

* patches test_program_bpf_instruction_introspection

* hard codes demote_sysvar_write_locks to false for serialization/encoding methods

* Revert "hard codes demote_sysvar_write_locks to false for serialization/encoding methods"

This reverts commit ae3e2d2e777437bddd753933097a210dcbc1b1fc.

* change the hardcoded ones to demote_sysvar_write_locks=true

* Use data_as_mut_slice

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-03-30 10:05:09 -07:00
sakridge 5e5b63712b
Status cache improvements (#16174) 2021-03-28 19:10:14 -07:00
Jeff Washington (jwash) 66c42f62d8
make AccountSharedData.data private to abstract storage (#16091)
* format

* 2 more tests

* use
2021-03-25 11:04:20 -05:00
Ryo Onodera 6d5c6c17c5
Simplify account.rent_epoch handling for sysvar rent (#16049)
* Add some code for special local testing

* Add comment to store_account_and_update_capitalization

* Simplify account.rent_epoch handling for sysvar rent

* Introduce *_for_test functions

* Add deprecation messages to existing api
2021-03-25 15:23:20 +09:00
sakridge 26af2d4c26
Up store limit for test_account_grow (#16099)
There can be more than 5 stores
2021-03-24 15:24:17 -07:00
Jeff Washington (jwash) 2d24d13046
AccountsSharedData: data copy on write (#15800)
* Arc<AccountData>

* try custom serializer

* adapt test from Behzad's change

* clippy

* simplify serialization

* remove abi example derive

* refactor 'take'

* remove serialization

* remove serialize calls

* remove account_data

* remove intos

* remove left over file
2021-03-23 15:19:31 -05:00
Jeff Washington (jwash) fde43a906d
revert test to use Account for serialization (#16063) 2021-03-22 22:48:09 +00:00
Jeff Washington (jwash) 4beb39f7a1
multi-pass bin scanning (#15377)
* multi-pass bin scanning

* pr feedback

* format

* fix typo

* adjust metrics for code changes

* merge errors
2021-03-18 10:32:07 -05:00
carllin f548a04fae
Allow unbounded wallclock processing time in tests (#15961) 2021-03-17 15:48:50 -07:00
DimAn a5d144b00f Revert to removing only tmp- 2021-03-17 10:31:01 -07:00
DimAn 0209d334bd Add option for separate snapshot location
(cherry picked from commit 6126878f509c69e23480a5ec22b3271e2b16e072)
2021-03-17 10:31:01 -07:00
Michael Vines a2eb655322 =1.7.0 2021-03-16 07:51:07 +00:00
Jeff Washington (jwash) efee8b62d7
a few missed set_data calls (#15846)
* a few missed set_data calls

* another set data call
2021-03-15 21:57:23 -05:00
Jeff Washington (jwash) e35d738906
rework BankClient.get_account_data (#15853) 2021-03-15 19:27:53 -05:00
carllin c1ba265dd9
Wallclock BankingStage Throttle (#15731) 2021-03-15 17:11:15 -07:00
Michael Vines 0c9ca5522c Bump version to v1.7.0 2021-03-13 09:01:21 +00:00
Justin Starry 918d04e3f0
Add more slot update notifications (#15734)
* Add more slot update notifications

* fix merge

* Address feedback and add integration test

* switch to datapoint

* remove unused shred method

* fix clippy

* new thread for rpc completed slots

* remove extra constant

* fixes

* rely on channel closing

* fix check
2021-03-12 21:44:06 +08:00
Ryo Onodera 4bbeb9c033
Remove old feature: simple_capitalization (#15763)
* Remove old feature: simple_capitalization

* Fix another failing test in core

* Finish up test cleanup

* Further clean up a bit
2021-03-12 11:12:40 +09:00
Jeff Washington (jwash) 952c3bcbb7
AccountSharedData construction (#15790) 2021-03-11 18:09:04 -06:00
Jeff Washington (jwash) 3419a5446e
AccountSharedData.set_data (#15781)
* account.set_data and resize_data

* remove data_resize
2021-03-11 16:40:45 -06:00
Michael Vines 1061d021c9 Unpack snapshot AppendVecs directly into account paths 2021-03-11 08:25:24 -08:00
Trent Nelson 24d18b3cf2 docs: add docs links for crates published to crates.io 2021-03-11 08:38:18 +00:00
Jack May 478518308d
Share RO and Executable accounts within invocations (#15799) 2021-03-10 23:04:00 -08:00
Michael Vines 65dd177601 AppendVec cleanup: remove dead code, rename non-idiomatic "new" function 2021-03-10 14:50:59 -08:00
Jeff Washington (jwash) 1135ffd595
mut data refs as slice (#15782) 2021-03-10 15:28:03 -06:00
Tyera Eulberg 9c1198c0c7
Improve load_largest_accounts more (#15785)
* Add load_largest_accounts bench

* Check lamports before address filter

* Use BinaryHeap, add Accounts test

* Use pubkey reference in the min-heap

Also, flatten code with early returns

Co-authored-by: Greg Fitzgerald <greg@solana.com>
2021-03-10 11:22:02 -07:00
Jack May 369e13b111
cleanup old runtime features (#15787) 2021-03-10 09:48:41 -08:00
Tyera Eulberg 3355efbb5a
Fix test to use AccountSharedData 2021-03-09 20:47:54 -07:00
Tyera Eulberg 5991cef5f5
Improve load_largest_accounts and add Bank test (#15775) 2021-03-10 00:23:41 +00:00
Jeff Washington (jwash) 52e54e1100
account.data -> data() (#15778) 2021-03-09 22:31:33 +00:00
Jeff Washington (jwash) 8a3135d17b
Account->AccountSharedData (#15691) 2021-03-09 15:06:07 -06:00
Tyera Eulberg e3e8179f2d
Sdk: start to appease clippy's integer arithmetic check (#15736)
* Move to module-level clippy allowance

* Fix stake_weighted_timestamp math

* Fix genesis_config

* Fix shred_version

* Fix hard_forks

* Fix process_instruction

* Add ArithmeticOverflow ix error

* Fix nonce_keyed_account

* Update BankSlotDelta frozen abi due to new ix error
2021-03-08 18:37:57 -07:00
Ryo Onodera beac2d6d3f Remove stale comment 2021-03-08 10:18:32 +09:00
Ryo Onodera 8b0c6db871
Remove old feature: cumulative_rent_related_fixes (#15754) 2021-03-08 09:58:50 +09:00
sakridge c5371fdc85
More debug on shrink and flush (#15518) 2021-03-05 14:10:34 -08:00
Jeff Washington (jwash) 13b9bb6790
refactor function to store into account after data is finished (#15710) 2021-03-05 11:02:35 -06:00
Greg Fitzgerald edd159e7d7
Fix typos (#15721) 2021-03-05 07:16:53 +00:00
Tyera Eulberg 872f7117c3
Use OrderedIterator to produce TransactionLogInfo (#15712)
* Add failing test

* Fix iteration_order issue with stored logs
2021-03-04 15:26:19 -07:00
Michael Vines 1e2f5a5f55 Permit the snapshots/status_cache file to be sparse 2021-03-04 12:36:34 -08:00
Jack May b53cb8eb2d
Deprecate Instruction::new (#15695) 2021-03-04 05:46:48 +00:00
Jeff Washington (jwash) be35c1c1b7
add execute detail timings (#15638) 2021-03-03 17:07:45 -06:00
Greg Fitzgerald 9bbe015a05
Document and clean up AppendVec (#15640)
* Document AppendVec

* Remove the almost-duplicate state in AppendVec

AppendVec was maintaining two offsets, `current_len` and
`append_offset`. Despite the different looking names, the two
values have the same meaning, but were updated at slightly different
times. When appending a batch of accounts, `current_len` updates would
be immediately available to other threads after each append, whereas
`append_offset` would only be updated after its mutex was unlocked.
`append_offset` is redundant. By removing it, we eliminate potential
bugs and no longer need to suppress clippy warnings.

* Remove get_mut() from AppendVec design

Only the offset into the AppendVec memory is thread-safe. The memory
itself is only threadsafe because it is append-only and is otherwise
unprotected. Adding get_mut() would only be safe if the memory was
protected by a ReadWrite lock.
2021-03-03 12:51:48 -08:00
Jack May a9c8dbfd0c
Add error reporting to system program (#15644) 2021-03-02 21:44:25 -08:00
Jeff Washington (jwash) a2c47db920
extract expected cluster type (#15615)
* extract expected cluster type

* fix clippy warning

* more clippy
2021-03-02 13:55:34 -06:00
Greg Fitzgerald 43663b1750
Fix clippy warnings (#15632) 2021-03-02 08:58:14 -08:00
Jeff Washington (jwash) da47ab7e50
fix indentation format problem (#15618)
* fix indentation format problem

* fix indentation format problem
2021-03-02 10:46:00 -06:00
Greg Fitzgerald aad79d9d5c
Remove unused Accounts state (#15611)
* Remove dead state from Accounts

* Refactor to remove reference to self.slot

Account::default() has zero lamports, so would return None anyway
2021-03-02 08:28:25 -08:00
Greg Fitzgerald 2463cc1e6a
Fix typos (#15610) 2021-03-02 06:36:49 -08:00
carllin 97eaf3c334
Fix finalize_dead_slot_removal() of cached slots decrementing refcount (#15534) 2021-02-26 17:49:37 -08:00
sakridge 8399851d11
check program owners (#15495)
* check program owners

* BankSlotDelta should change because InstructionError variant added

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-02-26 22:21:34 +00:00
Ryo Onodera 21b43009f6
Introduce ttl eviction for RecycleStore (#15513) 2021-02-25 17:27:27 +09:00
Tyera Eulberg eddb7f98f5
Check vote account initialization (#15503)
* Check account data_len on Vote account init

* Check account data populated on update_cached_accounts
2021-02-24 10:00:48 -07:00
Jack May 2f46da346d
gracefully handle vote account without authorized voter (#15501) 2021-02-24 00:00:24 -08:00
carllin 9a7cd8885d
Remove read only locks when they hit ref count zero, cleanup accounts locking (#15449) 2021-02-22 23:56:43 -08:00
Michael Vines 5df36aec7d Pacify clippy 2021-02-19 20:08:41 -08:00
Michael Vines 5d31763a3b Update ABI digest 2021-02-19 10:54:39 -08:00
Jack May 02432a548f
Cleanup old features (#15391) 2021-02-18 09:56:11 -08:00
Jeff Washington (jwash) e3006e3b46
correctly reserve space for pre_accounts vector (#15378) 2021-02-18 09:37:24 -06:00
Jon Cinque 0f6f6080f3
sdk: Add Borsh support for types and utilities (#15290)
* sdk: Add Borsh to Pubkey

* Add serialization error for easier borsh integration

* Add Borsh usage to banks-client and sdk

* Rename SerializationError -> IOError

* Add new errors to proto

* Update Cargo lock

* Update Cargo.lock based on CI

* Clippy

* Update ABI on bank

* Address review feedback

* Update sanity program instruction count test
2021-02-18 11:14:56 +01:00
Trent Nelson 7f7370c306 Re-allow clippy::integer_arithmetic at crate-level 2021-02-17 13:55:08 -07:00
Michael Vines 9ba69a7381 Adapt to fs_extra 1.2.0 2021-02-17 12:44:58 -08:00
Jeff Washington (jwash) 7fee0bc69c
move hashing functionality to accounts_hash (#15353) 2021-02-17 00:29:50 +00:00
Jeff Washington (jwash) 8367740ff9
compute merkle root on chunks of fanout^3 (#15344)
* compute merkle root on chunks of fanout^3

* improve test_accountsdb_compute_merkle_root_large
2021-02-16 17:03:35 -06:00
Jeff Washington (jwash) ba02452d75
add validator flag no-accounts-db-index-hashing (#15350)
* add validator flag no_accounts_db_index_hashing

* add validator flag no_accounts_db_index_hashing
2021-02-16 21:13:48 +00:00
Jeff Washington (jwash) d83aeebcca
change 2d cumulative calculation (#15346) 2021-02-16 14:24:45 -06:00
Jeff Washington (jwash) 708b8380b7
assert 2 items in stores don't have same slot and version with different values (#15250)
* assert 2 items in stores don't have same slot and version with different values

* add message to assert
2021-02-16 18:12:59 +00:00
Jeff Washington (jwash) 53959b4bbc
divide account scan into bins for more efficient downstream processing (#15303) 2021-02-15 21:27:39 -06:00
Jeff Washington (jwash) 78a176fc00
rename CumulativeOffset in preparation of 2d support (#15338) 2021-02-15 20:57:05 -06:00
Ryo Onodera 30f18319f2
Track RecycleStore basic stats with needed refactor (#15291)
* Track RecycleStore basic stats with needed refactor

* Fix another wrong metrics def
2021-02-15 16:09:17 +09:00
Jeff Washington (jwash) 866ce236b1
add another hash check (#15278)
* add another hash check

* update comment
2021-02-12 18:35:11 -06:00
Jeff Washington (jwash) b8448f4189
merkle root code no longer adds lamports (#15298) 2021-02-12 17:30:14 -06:00
Michael Vines 2e7aebf0bb Rework spl_token_v2_self_transfer_fix to avoid any SPL Token downtime 2021-02-12 22:46:39 +00:00
Jeff Washington (jwash) 8944efddf7
combine flatten and hash (#15269) 2021-02-12 11:35:54 -06:00
Jeff Washington (jwash) eced3776b0
add metrics to hash calculation and consolidate (#15165)
* add metrics to hash calculation

* add metrics to hash calculation

* create HashStats struct to consolidate hash timing

* formatting
2021-02-11 20:06:04 -06:00
Jeff Washington (jwash) eb54e544c2
add div_ceil (#15267) 2021-02-11 20:05:41 -06:00
Jeff Washington (jwash) 59e55f0371
remove hash accumulation with lamports (#15275) 2021-02-11 20:05:22 -06:00
Michael Vines 47c60f8e98 Clean up mainnet-beta inflation candidate features 2021-02-10 17:46:22 -08:00
Jeff Washington (jwash) e59a24d9f9
pass expected capitalization to hash calculation to improve assert msg (#15191)
* cleanup if

* pass expected capitalization to hash calculation to improve assert message

* fix bank function

* one more level

* calculate_accounts_hash_helper

* add slot to error message

* success
2021-02-10 14:38:00 -06:00
Tyera Eulberg da6753b8c0
Warp timestamp and extend max-allowable-drift for accommodate slow blocks (#15204)
* Remove timestamp_correction feature gating

* Remove timestamp_bounding feature gating

* Remove unused deprecated ledger code

* Remove unused deprecated unbounded-timestamp code

* Enable independent adjustment of fast/slow timestamp bounding

* Update timestamp bounds to 25% fast, 80% slow; warp timestamp

* Update bank hash test

* Add PR number to feature

Co-authored-by: Michael Vines <mvines@gmail.com>

Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-09 15:49:00 -07:00
Jeff Washington (jwash) 8424fe2c12
use index version of calculating hash (#15189)
* use index version of calculating hash

* invert const

* formatting
2021-02-09 10:37:50 -06:00
Jeff Washington (jwash) 600cea274d
change store account scan to not use dashmap (#15104)
* change store account scan to not use dashmap

* add test_accountsdb_de_dup_accounts_from_stores

* add tests

* add test_accountsdb_flatten_hash_intermediate

* add tests

* add sort test

* add test

* clippy

* first_slice -> is_first_slice

* comment

* use partial_cmp
2021-02-05 21:16:30 -06:00
Jeff Washington (jwash) fbf9dc47e9
Threadpool2 (#15151)
* rework thread pool for hash calculation

* rename
2021-02-05 18:48:16 -06:00
Jeff Washington (jwash) fabecdc86c
use thread pool for non-index hash calculations (#15149) 2021-02-05 19:48:55 +00:00
behzad nouri 6fd5ec0e4c
caches descendants in bank forks (#15107) 2021-02-05 18:00:45 +00:00
Jeff Washington (jwash) f85be6259b
sentinel value for zero lamport accounts in hash scanning (#15097)
* sentinel value for zero lamport accounts in hash scanning

* fix test
2021-02-05 11:23:48 -06:00
Ryo Onodera c69027e5aa
chore: bump serde from 1.0.118 to 1.0.122 (#15126)
* chore: bump serde from 1.0.118 to 1.0.122

* Update programs/bpf/Cargo.lock
2021-02-05 19:23:24 +09:00
Jeff Washington (jwash) 7d9f5ad525
remove unused arg from function (#15096) 2021-02-04 20:46:30 -06:00
Jeff Washington (jwash) f0d58f5549
move timer end outside if (#15087) 2021-02-04 20:43:23 -06:00
carllin e4d0d4bfae
Fix pubkey refcount for shrink + clean (#14987) 2021-02-04 12:44:19 -08:00
Jeff Washington (jwash) f49a70e626
borrow storages (#15088) 2021-02-04 11:29:03 -06:00
sakridge e5225b7e68
Add ref count from storage (#15078) 2021-02-04 09:11:05 -08:00
Jeff Washington (jwash) e7f53182e2
fix merge conflict (#15085) 2021-02-04 09:20:50 -06:00
Jeff Washington (jwash) 600ff0d915
calculate hash from store instead of index (#15034)
* calculate hash from store instead of index

* restore update hash in abs
2021-02-04 09:00:33 -06:00
carllin 2970b59853
Don't load all accounts into memory for capitalization check (#14957)
* Don't load all accounts into memory for capitalization check
2021-02-03 15:00:42 -08:00
Jon Cinque 8e93a784f3
program-test: Add ability to warp to the future (#14998)
* program-test: Add ability to warp to the future

* Make `start_local_server` take by value

* Remove clear_invoke_context
2021-02-03 23:31:36 +01:00
sakridge 2b9a0311d3
Don't reset accounts if the remove_account comes from a clean (#15022)
Store may be in-use with a snapshot creation, so don't disturb
it's state.
2021-02-03 13:06:28 -08:00
sakridge 37aac5a12d
Cleanup v1 shrink path (#15009)
move legacy functions to another impl block
2021-02-02 13:29:46 -08:00
Jeff Washington (jwash) 22800054e6
streamline calculate_accounts_hash (#14980) 2021-02-02 10:58:19 -06:00
Ryo Onodera 31168fe343
Disable AppendVec warn! for now (#14996)
* Disable AppendVed warn! for now

* Fix version...

* Update append_vec.rs
2021-02-02 14:52:30 +00:00
dependabot[bot] 1df93fa2be
chore: bump serde from 1.0.112 to 1.0.118 (#14828)
* chore: bump serde from 1.0.112 to 1.0.122

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.112 to 1.0.122.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.122)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* Update frozen_abi digest following serde update

* Revert "chore: bump serde from 1.0.112 to 1.0.122"

This reverts commit a3ef4442a4c985144ae2bd7ceaf8899a7ab8d7c0.

* Revert "[auto-commit] Update all Cargo lock files"

This reverts commit c41c3b005fb1ccade55155302c52cd5736c4b55f.

* chore: bump serde from 1.0.112 to 1.0.118

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.112 to 1.0.118.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.118)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* Remove serum-dex pinning

* blind commit!

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-02-02 23:28:16 +09:00
Jack May 73d9186502
More rich runtime logging (#14938) 2021-02-01 11:40:49 -08:00
sakridge 63c44bd690
Use helper for count() in accountsDB (#14953) 2021-01-31 09:32:02 -08:00
Jack May 07cef5a557
Buffer authority must match upgrade authority for deploys and upgrades (#14923) 2021-01-29 12:43:42 -08:00
sakridge 65315fa4c2
Use already-generated key set to populate dirty keys for clean (#14905)
Don't need to scan the stores again when we already found the key
set of updates per slot. Just insert it earlier.
2021-01-29 11:44:43 -08:00
Trent Nelson 85b5dbead6 Add feature for pending SPL Token self-transfer fix 2021-01-29 04:19:33 +00:00
Jack May 0b1015f7d3
Richer runtime failure logging (#14875) 2021-01-28 10:04:54 -08:00
sakridge a53b8558cd
Only mmap file from snapshot once (#14815) 2021-01-28 08:15:33 -08:00
Jack May 63429507b2
Add syscall feature activation test (#14890) 2021-01-27 17:21:25 -08:00
carllin 72f10f5f29
Aggregate purge and shrink metrics (#14763)
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-27 01:39:47 -08:00
carllin daddcd361a
Snapshots missing slots from accounts cache clean optimization (#14852)
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-26 20:08:55 -08:00
Tyera Eulberg ffa5c7dcc8
Deprecate commitment variants (#14797)
* Deprecate commitment variants

* Add new CommitmentConfig builders

* Add helpers to avoid allowing deprecated variants

* Remove deprecated transaction-status code

* Include new commitment variants in runtime commitment; allow deprecated as long as old variants persist

* Remove deprecated banks code

* Remove deprecated variants in core; allow deprecated in rpc/rpc-subscriptions for now

* Heavier hand with rpc/rpc-subscription commitment

* Remove deprecated variants from local-cluster

* Remove deprecated variants from various tools

* Remove deprecated variants from validator

* Update docs

* Remove deprecated client code

* Add new variants to cli; remove deprecated variants as possible

* Don't send new commitment variants to old clusters

* Retain deprecated method in test_validator_saves_tower

* Fix clippy matches! suggestion for BPF solana-sdk legacy compile test

* Refactor node version check to handle commitment variants and transaction encoding

* Hide deprecated variants from cli help

* Add cli App comments
2021-01-26 19:23:07 +00:00
Michael Vines 2b50433099 Remove legacy_stake program 2021-01-25 19:34:40 -08:00
Jeff Washington (jwash) 1b85114a9c
remove legacy merkle root (#14772)
* remove legacy merkle root
f78197a

* clippy

* compile error

* borrow error

* derministic results

* clippy

* borrow
2021-01-25 15:26:08 -06:00
Alexander Meißner 5700e835ef
Refactors tuple of TransactionLoadResult into a struct. (#14773) 2021-01-25 10:35:08 +01:00
sakridge 0d32a0e0f4
Partial clean (#14800)
* Revert "Revert "Partial accounts clean (#14652)" (#14777)"

This reverts commit ad2e10e17b.

* Remove squashed uncleaned keys
2021-01-24 09:50:19 -08:00
Michael Vines cbffab7850 Upgrade to Rust v1.49.0 2021-01-23 19:16:36 -08:00
sakridge 96b27ca0cb
Remove zero_lamport_pubkeys reference (#14808) 2021-01-23 18:31:55 +00:00
sakridge 424bb797a6
Speed up generate_index (#14792) 2021-01-23 08:05:05 -08:00
carllin c77461e428
Remove unnecesary flushes in previous roots (#14596)
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-23 04:02:44 -08:00
Michael Vines bf1943e489 Add solana-test-validator --warp-slot argument 2021-01-22 21:17:02 -08:00
Michael Vines dd5a2ef05f Add convenience function to create a snapshot archive out of any Bank 2021-01-22 21:17:02 -08:00
sakridge ad2e10e17b
Revert "Partial accounts clean (#14652)" (#14777)
This reverts commit ca7914aa3d.
2021-01-22 18:05:49 -08:00
Jack May 77572a7c53
Track account writable deescalation (#14626) 2021-01-22 15:28:01 -08:00
Jeff Washington (jwash) 18bd0c9a5b
speed up merkle calculation (#14710) 2021-01-22 09:21:50 -06:00
sakridge ca7914aa3d
Partial accounts clean (#14652)
Clean less keys by tracking the two cases:
* Touched keys per slot in uncleaned_keys derived from
accounts delta hash operation.
* Set of keys with any zero-lamport updates.
2021-01-21 11:01:53 -08:00
Michael Vines 072e5e54d8 Add generalized voting process to enable full inflation 2021-01-20 23:37:21 -08:00
carllin 2745b79b74
Parallel cache scan (#14544)
* Parallel cache scan

* PR comments

* PR comments

Co-authored-by: Carl Lin <carl@solana.com>
2021-01-20 08:50:17 +00:00
Jack May e3bd9e5300
Prevent the invoke and upgrade of programs in the same tx batch (#14653)
* Prevent the invoke and upgrade of programs in the same tx batch

* Pass program address as writable in the upgrade instruction
2021-01-19 16:24:44 -08:00
carllin 5f14f45282
More generic accounts purge functions (#14595)
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-17 20:31:03 -08:00
Jack May 66b54b852d
Update-executable flag in pre-accounts (#14622) 2021-01-15 17:48:29 -08:00
dependabot[bot] 4ecc0f4fa6
chore: bump rayon from 1.4.1 to 1.5.0 (#14177)
* chore: bump rayon from 1.4.1 to 1.5.0

Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.4.1 to 1.5.0.
- [Release notes](https://github.com/rayon-rs/rayon/releases)
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md)
- [Commits](https://github.com/rayon-rs/rayon/compare/rayon-core-v1.4.1...rayon-core-v1.5.0)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-01-15 18:07:35 +09:00
sakridge 907f518f6d
Add load/execute/store timings (#14561) 2021-01-14 14:14:16 -08:00
Jack May 6e8a1ba7de
Load executable accounts from invoke context (#14574) 2021-01-14 00:19:22 -08:00
carllin 6dfad0652f
Cache account stores, flush from AccountsBackgroundService (#13140) 2021-01-11 17:00:23 -08:00
Michael Vines 4a66e3eddc Use standard tmp-snapshot- file prefix for the "new_state" archive for better cleanup/consistency 2021-01-11 16:51:26 -08:00
Tyera Eulberg b0e6e29527
Update timestamp max allowable drift to 50% of PoH (#14531)
* Repurpose warp-timestamp feature for general bump

* Change max_allowable_drift to 50%

* Fill in PR#

* Fix rpc test setup
2021-01-11 23:27:30 +00:00
Jack May 8ad5931bfc
Check native account owner (#14535) 2021-01-11 22:36:52 +00:00
Michael Vines 872a249e97 Restore snapshot hard linking 2021-01-11 12:11:25 -08:00
Michael Vines a95675a7ce
Avoid tmp snapshot backlog in SnapshotPackagerService under high load (#14516) 2021-01-11 10:21:15 -08:00
Michael Vines 7be6770808 Rename CompressionType to ArchiveFormat 2021-01-09 09:07:49 -08:00
Michael Vines 141e6706e6 Rename AccountsPackage::root to AccountsPackage::slot 2021-01-09 09:07:49 -08:00
Michael Vines 9f70f7dc3e Reduce TempDir exposure 2021-01-09 09:07:49 -08:00
Michael Vines f2a7f561a0 Replace incorrect symlink_dir usage with symlink_file 2021-01-09 09:07:49 -08:00
Michael Vines fe0ba4a429 Create account paths once 2021-01-09 09:07:49 -08:00
Jack May 58487c6360
Add buffer authority to upgradeable loader (#14482) 2021-01-08 09:37:57 -08:00
Ryo Onodera c9df6134fa
Save 7G mem on mainnet fixing AccIndex overalloc. (#14435) 2021-01-06 09:48:59 +09:00
Jack May a8b5a32b50
Gate cpi program account passing (#14443) 2021-01-05 21:53:41 +00:00
sakridge 53d65009a0
Lower recycle store count (#14429)
Too many stores can cause swap usage which
is detrimental to account store times.
2021-01-05 12:21:17 -08:00
Jack May 0619805806
Upgradeable programs needs program account's address as program id (#14417) 2021-01-04 21:45:05 +00:00
Michael Vines db6ee289c9 Don't bother restoring snapshots, they're never used 2021-01-02 17:26:33 -08:00
Michael Vines b6dcdb90e8 Remove dead code 2021-01-02 17:26:33 -08:00
carllin 5affd8aa72
Add secondary indexes (#14212) 2020-12-31 18:06:03 -08:00
sakridge 1c61d005b5
Re-enable shrink test (#14331) 2020-12-30 08:25:45 -08:00
Ryo Onodera 7893e2e307
Remove Testnet-specific old code (#14305) 2020-12-29 05:11:46 +09:00
Ryo Onodera addffd7694
Log error from AppendVec removal & a panic clean (#14302) 2020-12-29 04:59:09 +09:00
sakridge c693ffaa08
Fix subtraction overflow in metrics (#14290) 2020-12-27 16:26:22 -08:00
behzad nouri 49019c6613
obtains staked-nodes from the root-bank (#14257)
... as opposed to the working bank
2020-12-27 13:28:05 +00:00
Jack May ee0a80a092
Prevent bpf loader impersonators (#14278) 2020-12-24 03:04:48 +00:00
Jack May b1d702a618
Don't use caller passed executable account (#14276) 2020-12-23 14:34:14 -08:00
Jack May 5945305b1d
Save cloning program account data (#14251) 2020-12-23 10:18:14 -08:00
Trent Nelson 5b903318b2 vote: Add helper for creating current-versioned states 2020-12-22 19:37:26 -07:00
Trent Nelson 3881ae10fb Deinitialize nonce data upon zero balance 2020-12-22 19:37:26 -07:00
behzad nouri a14cfd660a
removes &Arc<Self> receivers (#14234) 2020-12-22 23:51:53 +00:00
Tyera Eulberg c276513b84
Ignore long/hanging test (#14261) 2020-12-22 16:13:03 -07:00
Tyera Eulberg 7042f11791
Feature-gate stake-program-v3 (#14232)
* Remove deprecated legacy stake program

* Add legacy stake program

* Strip out duplicative legacy code

* Feature-deploy stake-program-v3

* Add ownership check in stake processor
2020-12-22 11:20:38 -07:00
Jack May ab205b682a
Upgradeable programs called same as non-upgradeable (#14239)
* Upgradeable programs called same as non-upgradeable

* nudge
2020-12-22 09:26:55 -08:00
sakridge baa9602411
Add shrink paths (#14238) 2020-12-21 21:33:37 -08:00
behzad nouri 7b08cb1f0d
improves performance in replay-stage (#14217)
bank::vote_accounts returns a hash-map which is slow to iterate, but all uses
only require an iterator:
https://github.com/solana-labs/solana/blob/b3dc98856/runtime/src/bank.rs#L4300-L4306
Similarly, calculate_stake_weighted_timestamp takes a hash-map whereas it only
requires an iterator:
https://github.com/solana-labs/solana/blob/b3dc98856/sdk/src/stake_weighted_timestamp.rs#L21-L28
2020-12-21 19:18:19 +00:00
Michael Vines 93ae177503 Do not delete ALL other snapshots before downloading a new snapshot 2020-12-21 08:58:52 +00:00
Tyera Eulberg e15f95a36f
Fix timestamp handling on ledger warp (#14210)
* Reset timestamp for slot and epoch-start on warp

* Fix genesis timestamp metric source

* Remove check that timestamp > unix_timestamp_from_genesis

Default to previous timestamp, not genesis timestamp

* Move timestamp metrics to report even on warp

* Initialize slot 0 timestamps correctly

* Add feature gate to warp testnet timestamp

* Review suggestion: simplify warp-timestamp slot check
2020-12-20 21:36:56 +00:00
Jack May e8cc0bef6c
Add CPI support for upgradeable loader (#14193) 2020-12-17 15:39:49 -08:00
behzad nouri d6d76219b6
caches staked nodes computed from vote-accounts (#13929) 2020-12-17 21:22:50 +00:00
Michael Vines efc091e28a Add transactionCount field to GetEpochInfo 2020-12-17 12:29:50 -08:00
carllin 75e9e321de
Fix race between setting tick height and calculating accounts hash (#14101)
Co-authored-by: Carl Lin <carl@solana.com>
2020-12-15 12:45:40 -08:00
Michael Vines 1e977ac025 Bump version to v1.6.0 2020-12-15 18:28:04 +00:00
Jack May d513b0c4ca
Add Program loader/environment instruction errors (#14120) 2020-12-15 09:54:07 -08:00
sakridge 53fe0bb5eb
Remove regular expressions, speeds up untar (#14119) 2020-12-15 08:48:21 -08:00
sakridge 5294f70189
Remove some non-warnings (#14115) 2020-12-14 16:26:17 -08:00
Jack May 9e90394583
Upgradeable loader (#13689) 2020-12-14 15:35:10 -08:00
Michael Vines e907510a1a clippy 2020-12-14 13:11:50 -08:00
Michael Vines 7143aaa89b Clippy 2020-12-14 08:03:29 -08:00
Ryo Onodera de9ac43ebf
Simple cap by including sysvars and native programs (#13884)
* Simpler cap by including sysvars and native programs

* Fix tests

* Add comment

* revert some unrelated code

* Update test_bank_update_sysvar_account for cap.

* Test cap. for add_native_program using new helper

* Improve the cap adjustment with new tests

* Fix typo...

* Adjust test for improved code coverage

* Rename simpler_capitalization => simple_capitalization

* More rename and bonus commenting
2020-12-14 21:03:24 +09:00
Ryo Onodera 5788e75188
Shrink slots by sparseness of written data size (#14072) 2020-12-14 02:58:38 +09:00
dependabot[bot] 88d950e5a8
chore: bump log from 0.4.8 to 0.4.11 (#13691)
* chore: bump log from 0.4.8 to 0.4.11

Bumps [log](https://github.com/rust-lang/log) from 0.4.8 to 0.4.11.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/compare/0.4.8...0.4.11)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-12-13 13:48:23 +09:00
dependabot[bot] 8df9bcc784
chore: bump libc from 0.2.80 to 0.2.81 (#14092)
* chore: bump libc from 0.2.80 to 0.2.81

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.80 to 0.2.81.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.80...0.2.81)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2020-12-13 12:13:17 +09:00
carllin 55fc963595
Move slot cleanup to AccountsBackgroundService (#13911)
* Move bank drop to AccountsBackgroundService

* Send to ABS on drop instead, protects against other places banks are dropped

* Fix Abi

* test

Co-authored-by: Carl Lin <carl@solana.com>
2020-12-13 01:22:34 +00:00
Trent Nelson edff62af3d Switch to memmap2 fork 2020-12-11 23:44:03 +00:00
Tyera Eulberg d6eff3d62c
Filter out stake and vote accounts with incorrect owners (#14062)
* Add failing test

* Check stake/vote accounts for validity

* Feature gate change

* Add datapoint

* Add test realism
2020-12-11 12:21:25 -07:00
Ryo Onodera 2d3a337200
Increase another snapshot limit: UNPACKED_COUNT (#14071) 2020-12-11 11:01:22 +00:00
Ryo Onodera 7078a6ac61
Add more comments in Stakes::store for its subtlety (#14065)
* Add more comments in Stakes::store for its subtlety

* more comment tweak
2020-12-11 18:13:36 +09:00
Ryo Onodera 09bd412b13
Wrong rpc inflation rate (#14063)
* Fix wrong inflation calculation in rpc

* Reorder

* Fix test
2020-12-11 07:42:39 +00:00
Michael Vines 0a9ff1dc9d Initial solana-test-validator command-line program 2020-12-11 04:17:38 +00:00
Jack May 83fda2d972
Clean invoke in prep for native invoke (#14055) 2020-12-10 18:25:57 -08:00
Ryo Onodera 164b7895b3
Tiny add_native_program bug fixes with cleanups (#14042)
* Tiny add_native_program bug fixes with cleanups

* Fix typo
2020-12-11 11:03:31 +09:00
Jack May 7abd8084b6
Add get_minimum_balance_for_rent_excemption to Client (#14048) 2020-12-10 16:39:28 -08:00
Jack May 5ea80e673f
Fix priv escalation test (#14046) 2020-12-10 22:36:33 +00:00
sakridge 68109a46e0
Lower priority for hashing threads. (#14043) 2020-12-10 12:26:47 -08:00
Ryo Onodera 14e241be35
Increase snapshot file size limits (#14024)
* Increase snapshot file size limits

* Fix test

* Reorder code a bit
2020-12-10 02:46:33 +09:00
Ryo Onodera 28b014ccf4
Add tests for store_account_and_update_capitalization (#14008) 2020-12-09 00:46:28 +09:00
Jack May dca579851c
Cannot change owner if account is executable (#13977) 2020-12-07 11:37:07 -08:00
Alexander Meißner a706706572
Validator CLI option to enable just-in-time compilation of BPF (#13789)
* Adds a CLI option to the validator to enable just-in-time compilation of BPF.

* Refactoring to use bpf_loader_program instead of feature_set to pass JIT flag from the validator CLI to the executor.
2020-12-07 09:49:55 +01:00
Jack May 882d033233
Don't use stale executor cache (#13966) 2020-12-05 05:54:42 +00:00
carllin 5dceddd21d
Remove associated functions on AccountsDb, replace with methods (#13953)
Co-authored-by: Carl Lin <carl@solana.com>
2020-12-03 17:53:42 -08:00
Trent Nelson 17defbff13 Make `Accounts::is_non_loader_key()` a method on `Message` 2020-12-02 20:10:08 +00:00
Trent Nelson 404fc1570d runtime: Replace `HashAgeKind` with `NonceRollbackInfo` 2020-12-02 20:10:08 +00:00
Trent Nelson be7760caa1 accounts: Replace nonce_rollback unreachable block with descriptive panic 2020-12-02 20:10:08 +00:00
Trent Nelson 47af5933ca accounts: Don't assume fee-payer is the first account 2020-12-02 20:10:08 +00:00
Trent Nelson 8df2a4bac0 runtime: Factor out `DurableNoncePartial` finishing logic and add tests 2020-12-02 20:10:08 +00:00
Michael Vines 73111b005f Reduce the number of snapshots 2020-12-01 11:13:37 -08:00
Michael Vines 254790f8c8 Fully clean up temporary snapshot staging directories 2020-11-30 19:40:06 -08:00
behzad nouri e1793e5a13
caches vote-state de-serialized from vote accounts (#13795)
Gossip and other places repeatedly de-serialize vote-state stored in
vote accounts. Ideally the first de-serialization should cache the
result.

This commit adds new VoteAccount type which lazily de-serializes
VoteState from Account data and caches the result internally.

Serialize and Deserialize traits are manually implemented to match
existing code. So, despite changes to frozen_abi, this commit should be
backward compatible.
2020-11-30 17:18:33 +00:00
Jack May ce4304cc9a
Add extra checks to verify_and_update (#13848)
* Add extra checks to verify_and_update

* nudge
2020-11-29 10:06:43 +00:00
Ryo Onodera 6048342c57
Enhance ledger-tool for multi-epoch rewards (#13837)
* Support ledger-tool for multi-epoch rewards

* nits

* Ensure not to skip some records in csv
2020-11-27 04:20:47 +00:00
carllin 955b99cf69
Fix roots_and_ref_count (#13827)
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-26 13:09:20 -08:00
Michael Vines 43b82b31e5 More TestValidator cleanup 2020-11-26 08:56:25 +00:00
Michael Vines b5f7e39be8 TestValidator public interface cleanup 2020-11-25 17:04:37 -08:00
Michael Vines 4ef2da0ff0 Add `solana logs` command 2020-11-25 11:44:41 -08:00
sakridge b70abdc645
Nonce updates (#13799)
* runtime: Add `FeeCalculator` resolution method to `HashAgeKind`

* runtime: Plumb fee-collected accounts for failed nonce tx rollback

* runtime: Use fee-collected nonce/fee account for nonced TX error rollback

* runtime: Add test for failed nonced TX accounts rollback

* Fee payer test

* fixup: replace nonce account when it pays the fee

* fixup: nonce fee-payer collect test

* fixup: fixup: clippy/fmt for replace...

* runtime: Test for `HashAgeKind::fee_calculator()`

* Clippy

Co-authored-by: Trent Nelson <trent@solana.com>
2020-11-24 23:53:51 -08:00
Tyera Eulberg c930f92411
Restore bank feature used to deprecate Rewards sysvar (#13786) 2020-11-24 10:22:31 -07:00
Michael Vines 60ce6f1a9b Truncate logs immediately upon collection. Also reduce limit from 100k to 10k 2020-11-23 17:20:02 -08:00
carllin 791fb17437
Prevent scans on unrooted slots from seeing partial clean (#13628)
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-20 13:01:04 -08:00
Michael Vines 62fa8b0ed8 Demote bank-process_transactions-error counters to info! 2020-11-20 10:43:07 -08:00
Tyera Eulberg c75d97e3f2
Add new inflation feature-ids (#13671)
* Add new inflation feature-ids, and full_inflation default values

* Compute inflation start from full_inflation activation

* Include pico_inflation in inflation start computation

* Add full-inflation constructor

* Align inflation taper with rewards accrual start and catch overflow edge case
2020-11-20 09:54:41 -07:00
Michael Vines 11e92f0c9f Snapshot archives are now cleaned up when `--snapshot-compression none` is used 2020-11-20 08:41:17 -08:00
Ryo Onodera b74d7b5758
Fix fragile tests in prep of stake rewrite pr (#13654)
* Fix fragile tests in prep of stake rewrite pr

* Restore BOOTSTRAP_VALIDATOR_LAMPORTS where appropriate

* Further clean up

* Further clean up

* Aligh with other call site change

* Remove false warn!

* fix ci!
2020-11-20 17:21:03 +09:00
behzad nouri a8c29505f0
sanitizes bloom filters to avoid division by zero (#13714)
Pull requests received over the wire can cause a validator to panic
because of division by zero in bloom filters:
https://github.com/solana-labs/solana/blob/af08ba93e/runtime/src/bloom.rs#L86-L88
2020-11-19 23:35:22 +00:00
Ryo Onodera 397cf726fc
Allow GNUSparse for genesis.bin (#13704) 2020-11-19 20:29:19 +00:00
Ryo Onodera 43d5e47ea9
Rewrite stake accounts for clear migration (#13461)
* Reduce overage stake by rewritng stake accounts

* Write tests and finish implemention

* Create and use new feature gate

* Clean up logging

* Fix typo

* Simplify enable_rewrite_stake

* Fix typo...

* Even simplify gating

* Add metrics
2020-11-20 05:15:06 +09:00
carllin 110acd20dc
Check for overflow in rent partition calculation (#13569)
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-19 04:17:00 -08:00
Tyera Eulberg 645598e615
Improve monitoring of timestamp correction and bounding (#13656)
* Include rejuvenated field in Clock parsing

* Expand timestamp-correction logging info
2020-11-17 21:51:43 +00:00
carllin 6276360468
Prevent scans from seeing root updates/clean (#13464)
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-16 17:23:11 -08:00
Ryo Onodera bcd303a447
ledger-tool cap: delegation owner and stake v2 flag (#13602)
* Output delegation owner as well

* Add --enable-stake-program-v2

* Small cleanup and add sanity assertion

* Fix typo...
2020-11-15 19:38:46 +00:00
Michael Vines baa6b3a261 Add stable program logging for BPF and native programs 2020-11-14 08:26:01 -08:00
Ryo Onodera c97a7d1105
Small cleaning update_epoch_stakes (#13576) 2020-11-13 11:16:38 +00:00
joshua-maros a8a77614fe
Fix potential undefined behavior (#13555)
* Switch to ouroboros 0.5.1

* Update other lock files
2020-11-13 18:12:41 +09:00
Ryo Onodera 89b474e192
Fix slow/stuck unstaking due to toggling in epoch (#13501)
* Fix slow/stuck unstaking due to toggling in epoch

* nits

* nits

* Add stake_program_v2 feature status check to cli

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-11-11 14:11:57 -07:00
carllin 2c2432fddc
Refactor function (#13294)
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-10 17:20:35 -08:00
Tyera Eulberg 91f4e99b4c
Fix parsing CreateAccountWithSeed instructions (#13513)
* Reduce required num_system_accounts and handle 2-account instructions properly

* Update CreateAccountWithSeed account docs to be correct

* Add CreateAccountWithSeed test
2020-11-10 23:51:53 +00:00
Tyera Eulberg c0e2ef06dc
Fix Bank accounts hash mismatch related to Clock::unix_timestamp (#13477)
* Test for different ancestors with mismatch bank hash

* Test cleanup

* Remove nondeterministic ancestor check

* Update timestamp bounding feature key

* Update design doc

* Filter recent_timestamps to nodes voting within the last epoch

Co-authored-by: Stephen Akridge <sakridge@gmail.com>
2020-11-10 02:10:09 +00:00
Ryo Onodera a81e7e7749
Output more inflation calc details in ledger-tool (#13345)
* Output more inflation calc details in ledger-tool

* Fix broken ci...

* Rename confusing variables

* Fix panic by wrapping PointValue with Opiton...

* Minor modifications

* Remove explict needless flush; Drop already does

* Yet another csv field adjustments

* Add data_size and rename epochs to earned_epochs

* Introduce null_tracer

* Unwrap Option in new_from_parent_with_tracer

* Don't shorten identifiers

* Allow irrefutable_let_patterns temporalily

* More null_tracer

* More field adjustments
2020-11-08 16:43:50 +09:00
Tyera Eulberg 1b1d9f6b0c
Feature-gate stake program (#13394)
* Add legacy stake-program handling

* Strip out duplicative legacy code

* Add feature for stake-program-fix

* Feature-deploy new stake program

* Expand comment
2020-11-06 08:08:11 +00:00
behzad nouri 8f0796436a
shares the lock on gossip when processing prune messages (#13339)
Processing prune messages acquires an exclusive lock on gossip:
https://github.com/solana-labs/solana/blob/55b0428ff/core/src/cluster_info.rs#L1824-L1825
This can be reduced to a shared lock if active-sets are changed to use
atomic bloom filters:
https://github.com/solana-labs/solana/blob/55b0428ff/core/src/crds_gossip_push.rs#L50
2020-11-05 15:42:00 +00:00
Tyera Eulberg bc62313c66
Allow feature builtins to overwrite existing builtins (#13403)
* Allow feature builtins to overwrite existing builtins

* Add feature_builtin ActivationType

* Correctly retain idempotent for replacing case

* Fix test

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-11-05 15:37:07 +00:00
Justin Starry c24fbb6f8b
Fix duplicate records of inner instructions (#13380)
* Fix duplicate records of inner instructions

* fix tests

* fix clippy

* Remove bad_inner_instructions
2020-11-05 15:23:52 +08:00
sakridge 43053dcc90
Re-use accounts stores (#12885)
* Re-use accounts_db stores

Creating files and dropping mmap areas can be expensive

* Add test for storage finder

Can encounter an infinite loop when the store is too small, but
smaller than the normal store size.

* Fix storage finding

* Check for strong_count == 1

* try_recycle helper
2020-11-04 09:17:05 -08:00
Ryo Onodera b0d1ae1d8b
Comment Stakes::clone_with_epoch (#13388) 2020-11-04 11:18:05 +00:00
Ryo Onodera 0e4509c497
Small code cleanup and typo fixes (#13325)
* Small code cleanup and typo fixes

* Clean up calculate_points_and_credits
2020-11-02 10:28:24 +09:00
sakridge 55b0428ff7
Add store-tool (#13254) 2020-11-01 09:41:28 -08:00
Michael Vines e60bda082d Adapt to feature::create_account 2020-11-01 05:53:29 +00:00
Tyera Eulberg 4260b3b416 Sample votes from ancestors back to root 2020-10-31 21:30:42 -07:00
Tyera Eulberg eb2560e782 Prevent block times from ever going backward 2020-10-31 21:30:42 -07:00
Tyera Eulberg 90778615f6 Use bounded timestamp-correction when feature enabled 2020-10-31 21:30:42 -07:00
Tyera Eulberg 80db6c0980 Add bounded timestamp-estimation method 2020-10-31 21:30:42 -07:00
Tyera Eulberg 0049ab69fb Repurpose unused as Clock::epoch_start_timestamp; add gated update 2020-10-31 21:30:42 -07:00
Michael Vines 4b65e32f22 Move Feature struct to solana-program 2020-10-30 17:57:51 -07:00
Ryo Onodera bc7133d752
Print the entry type as well when checking archive (#13312) 2020-10-31 01:36:12 +09:00
Michael Vines 9263ae1c60 Simplify CPI interface into MessageProcessor 2020-10-30 09:20:09 +00:00
Michael Vines da9548fd12 de-mut some InvokeContext methods 2020-10-30 09:20:09 +00:00
Michael Vines 7d686b72a0 Add Bank::set_bpf_compute_budget() 2020-10-29 21:45:24 -07:00
Michael Vines 33884d847a Remove programs clone() 2020-10-29 21:45:24 -07:00
Michael Vines 225bed11c7 Remove Entrypoint type 2020-10-29 21:45:24 -07:00
Michael Vines 2664a1f7ef Remove MessageProcessor::loaders 2020-10-29 21:45:24 -07:00
Michael Vines df8dab9d2b Native/builtin programs now receive an InvokeContext 2020-10-29 21:45:24 -07:00
Alexander Meißner 6606590b81
Updates rbpf to v0.2.0, (#12951)
which unifies the interfaces of the interpreter and the JIT.
However, the JIT is not enabled yet.
2020-10-29 11:34:52 -07:00
Alexander Meißner 65ee3a6bdd
Refactors the common code of test and bench targets into the solana_runtime::bpf_test_utils module. (#13203) 2020-10-29 10:04:47 +01:00
Jack May c458d4b213
move Account to solana-sdk (#13198) 2020-10-28 22:01:07 -07:00
sakridge d5e439037b
Make bad_bank_hash test faster (#13206) 2020-10-28 21:44:42 -07:00
Tyera Eulberg c2dbf53d76
Update FeatureSet::active to include slot-activated (#13256)
* Update FeatureSet::active to include slot-activated

* Clippy suggestion
2020-10-29 02:34:19 +00:00
Ryo Onodera 4698ee5e4a
Improve final report of ledger-tool capitalization (#13232) 2020-10-28 17:10:09 +00:00
Ryo Onodera 7d2962135d
Use pico inflation for ledger-tool capitalization --enable-inflation (#13215)
* Use pico inflation for ledger-tool capitalization --enable-inflation

* rust fmt
2020-10-28 15:30:39 +09:00
Justin Starry 631f029fe9
Account for forward delay in transaction simulation (#13199) 2020-10-27 17:25:08 +00:00
Michael Vines 1b343665a1 Move KeyedAccount out of solana-program. Native programs are not supported by solana-program 2020-10-26 18:54:54 -07:00
Greg Fitzgerald f58bc8589d
Add Bank::get_signature_status_with_blockhash() (#13167)
Get the signature status in O(1) time, instead of O(n) where
n is the number of blockhashes in the StatusCache.
2020-10-26 23:52:57 +00:00
Michael Vines a4956844bd Update frozen_abi hashes
The movement of files in sdk/ caused ABI hashes to change
2020-10-24 08:37:55 -07:00
sakridge 7d2729f6bd
shrink debug (#13089) 2020-10-23 08:05:19 -07:00
Michael Vines 959880db60 Remove unused pubkey::Pubkey imports 2020-10-21 19:08:13 -07:00
Michael Vines 17c391121a Run `codemod --extensions rs Hash::new_rand solana_sdk:#️⃣:new_rand` 2020-10-21 19:08:13 -07:00
Michael Vines 7bc073defe Run `codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand` 2020-10-21 19:08:13 -07:00
carllin e6b821c392
Finer grained AccountsIndex locking (#12787)
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-21 17:05:27 -07:00
carllin c8fc0a6ba1
Don't hold dashmap write lock in store create (#13007)
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-21 12:54:09 -07:00
Ryo Onodera 608b81b412
Port various rent fixes to runtime feature (#12842)
* Port various rent fixes to runtime feature

* Fix CI

* Use more consistent naming...
2020-10-21 22:19:25 +09:00
Michael Vines 6858950f76 Remove frozen ABI modules from solana-sdk 2020-10-20 16:11:30 -07:00
Ryo Onodera c0675968b1
Support Debug Bank (#13017) 2020-10-21 01:05:45 +09:00
sakridge 456eae6ccb
Calculate accounts hash async in accounts background service (#12852) 2020-10-19 09:48:29 -07:00
behzad nouri 48283161c3
passes through feature-set to gossip requests handling (#12878)
* passes through feature-set to down to gossip requests handling
* takes the feature-set from root_bank instead of working_bank
2020-10-15 20:54:21 +00:00
sakridge 4f6ebbdbe0
Add timings for store accounts (#12882) 2020-10-15 12:14:39 -07:00
Jack May 3f9e6a600b
program log pubkey as base58 (#12901) 2020-10-15 09:11:54 -07:00
Michael Vines 4b04ed86b6 Don't report RewardType::Fee when none was awarded 2020-10-14 09:06:20 -07:00
carllin f8d338c9cb
Switch accounts storage lock to DashMap (#12126)
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-13 18:29:50 -07:00
Ryo Onodera 7de7efe96c
Add nop feature set for upcoming ported rent fixes (#12841) 2020-10-13 16:34:49 +00:00
Jack May c24da1ee16
fix native_loader behavior for invalid accounts (#12814) 2020-10-12 13:40:04 -07:00
Eric Williams 1ab36203de
Retry inflation parameter update along with documentation updates (#12778)
* update initial inflation parameter

* update economics section of docs
2020-10-12 13:10:57 -07:00
Ryo Onodera 2f5bb7e507
Move no-0-rent rent dist. behavior under feature (#12804) 2020-10-12 08:47:52 +00:00
Michael Vines 8a119c1483 Bump reqwest/rayon to get past `cargo audit` 2020-10-11 18:51:09 +00:00
Michael Vines 1fc7c1ecee Don't bother paying 0 rent 2020-10-10 17:02:26 +00:00
Michael Vines c5c8da1ac0 Expose all rewards (fees, rent, voting and staking) in RPC getConfirmedBlock and the cli 2020-10-09 21:54:13 -07:00
Michael Vines 403790760c Default to 50% rent burn 2020-10-09 21:52:31 -07:00
Tyera Eulberg b028c47d2b
Correct Bank timestamp drift every slot (#12737)
* Move timestamp helper to sdk

* Add Bank method for getting timestamp estimate

* Return sysvar info from Bank::clock

* Add feature-gated timestamp correction

* Rename unix_timestamp method to be more descriptive

* Review comments

* Add timestamp metric
2020-10-09 21:53:41 +00:00
Jack May c3907be623
Add adjustable stack size and call depth (#12728) 2020-10-09 13:07:09 -07:00
carllin 16d45b8480
Fix zero-lamport accounts preventing slot cleanup (#12606)
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-09 12:40:08 -07:00
Jack May 2cd7cd3149
Bump max invoke depth to 4 (#12742) 2020-10-09 10:33:12 -07:00
carllin c879e7c1ad
Fix fee mismatch on snapshot deserialize (#12697)
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-08 23:44:41 -07:00
Michael Vines c8807d227a Add inflation_kill_switch feature 2020-10-08 21:31:31 -07:00
Josh 8f5431551e
Store program logs in blockstore / bigtable (TransactionWithStatusMeta) (#12678)
* introduce store program logs in blockstore / bigtable

* fix test, transaction logs created for successful transactions

* fix test for legacy bincode implementation around log_messages

* only api nodes should record logs

* truncate transaction logs to 100KB

* refactor log truncate for improved coverage
2020-10-08 12:06:15 -07:00
Michael Vines 11df2e2236 Bump version to v1.5.0 2020-10-08 04:51:36 +00:00
Jack May 973f0965e1
Add ristretto multiply syscall (#12699) 2020-10-06 23:52:13 -07:00
carllin 64c4861e04
Fix error in max root calculation (#12661)
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-03 15:18:58 -07:00
carllin adf6f748c0
Disable rent in flaky test (#12658)
* Disable rent in test

Co-authored-by: Carl Lin <carl@solana.com>
2020-10-03 12:44:03 -07:00
Jack May adeb06e550
Check CPI program is executable (#12644) 2020-10-02 13:55:22 -07:00
Ryo Onodera e3773d919c
Avoid overflow when computing rent distribution (#12112)
* Avoid overflow when computing rent distribution

* Use assert_eq!....

* Fix tests

* Add test

* Use FeatureSet

* Add comments

* Address review comments

* Tweak a bit.

* Fix fmt
2020-10-01 11:59:28 +09:00
Tyera Eulberg c31a34fbcb
Include post balance information for rewards (#12598)
* Include post balance information for rewards

* Add post-balance to stored Reward struct

* Handle extended Reward in bigtable

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-30 17:57:06 -06:00
Jack May 058bca6632
add sha256 syscall (#12569) 2020-09-29 23:29:20 -07:00
Trent Nelson 3c7b9c2938 Move remaining nonce utils from runtime to SDK 2020-09-30 05:45:42 +00:00
Trent Nelson 65b868f4eb Move nonced tx helpers to SDK 2020-09-30 05:45:42 +00:00
Trent Nelson caec631344 runtime: Move prepare_if_nonce_account into accounts 2020-09-30 05:45:42 +00:00
Michael Vines f57af4fec2 epoch_rewards datapoint now includes the correct rewards epoch (previous epoch) 2020-09-29 22:44:02 -07:00
Jack May 575400c83f
Run tests with features on rather then off (#12567) 2020-09-29 20:18:28 -07:00
Michael Vines 82848d6c73 Include active stake in 'epoch_rewards' datapoint 2020-09-29 16:50:10 -07:00
Jack May 74fcb184b2
Pipe FeatureSet though InvokeContext (#12536)
* Pipe FeatureSet though InvokeContext

* gate program size cap

* nit
2020-09-29 21:36:30 +00:00
Jack May 2ff983647f
Move process_instruction defs to runtime (#12507) 2020-09-29 01:36:46 -07:00
Tyera Eulberg 89621adca7
Rpc -> proper optimistic confirmation (#12514)
* Add service to track the most recent optimistically confirmed bank

* Plumb service into ClusterInfoVoteListener and ReplayStage

* Clean up test

* Use OptimisticallyConfirmedBank in RPC

* Remove superfluous notifications from RpcSubscriptions

* Use crossbeam to avoid mpsc recv_timeout panic

* Review comments

* Remove superfluous last_checked_slots, but pass in OptimisticallyConfirmedBank for complete correctness
2020-09-28 20:43:05 -06:00
carllin 06f84c65f1
Fix rooted accounts cleanup, simplify locking (#12194)
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-28 16:04:46 -07:00
Michael Vines 833ad20b01 Add Builtin AbiExample 2020-09-28 12:50:19 -07:00
Michael Vines 31696a1d72 Port BPFLoader2 activation to FeatureSet and rework built-in program activation 2020-09-28 12:50:19 -07:00
Michael Vines 2956cc5aed Add a couple feature tests 2020-09-28 12:24:09 -07:00
Michael Vines f9a74b51ef Add feature to resolve spl-token v2 multisig bug 2020-09-28 09:25:48 -07:00
Michael Vines aa5c008fa8 Add pico-inflation feature 2020-09-28 08:25:00 -07:00
Jack May 965f653471
Add copy-on-write executor cache (#12502)
* Add copy-on-write executor cache

* Add remove_executor function to the bank
2020-09-26 17:54:11 +00:00
Michael Vines a36252bfa0 Remove continues from compute_active_feature_set() 2020-09-26 06:10:50 +00:00
Michael Vines b15d6148dd Deerror 2020-09-25 22:18:43 -07:00
carllin 00b36e6db2
Fix ref-count for multiple stores to the same pubkey in a slot, fixes zero lamport purge detection (#12462)
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-25 18:54:48 -07:00
Michael Vines e50386f928 Add epoch rewards metric datapoint 2020-09-26 01:05:57 +00:00
Michael Vines 5d6410c1cb Port fix_recent_blockhashes_sysvar_delay to FeatureSet 2020-09-25 15:07:25 -07:00
Michael Vines 5dcf348098 Improve 'Failed to create snapshot archive' warning message 2020-09-25 20:02:49 +00:00
Michael Vines e39fac9f01 Rename active() to is_active() 2020-09-25 11:40:36 -07:00
Michael Vines 199940d683 cargo fmt 2020-09-25 11:40:36 -07:00
Michael Vines 92406cf9a0 FeatureSet test 2020-09-25 11:40:36 -07:00
Michael Vines cc6ba1e131 Remove id field 2020-09-25 11:40:36 -07:00
Michael Vines c10da16d7b Port instructions sysvar and secp256k1 program activation to FeatureSet 2020-09-25 11:40:36 -07:00