Commit Graph

1465 Commits

Author SHA1 Message Date
Jeff Washington (jwash) 6381ee38eb
reclaims unref accounts from index (#16838) 2021-04-28 08:50:38 -05:00
Jeff Washington (jwash) 1864bc2080
write Option<AccountSharedData> (#16874)
* write Option<&AccountSharedData>

* add comment
2021-04-28 08:47:26 -05:00
Brooks Prumo 1eaff394da
Refactor `collect_uncleaned_pubkeys_to_slot()` (#16879)
* Refactor `collect_uncleaned_pubkeys_to_slot()`

While working on another issue (#16580), I came across
`collect_unclean_pubkeys_to_slot()` and had difficulty understanding it.
Since the function does a few logically separate things, I split the
function up.  I also added documentation, removed an unused return value,
and renamed the functions to be more specific.

This commit is to split up an existing PR (#16786), where I had both this
aesthetic change _and_ a behavioral change.
2021-04-28 08:16:12 -05:00
Jeff Washington (jwash) 1bd623cd15
private AccountSharedData.rent_epoch (#16844) 2021-04-27 13:51:13 -05:00
Jeff Washington (jwash) 8f56c116d7
tests: lamports += to checked_add (#16842) 2021-04-27 10:23:07 -05:00
Jeff Washington (jwash) d533f77301
bank.withdraw uses checked_sub_lamports (#16848)
* bank.withdraw uses checked_sub_lamports

* retain previous codepath for error

* map_err from clippy
2021-04-27 09:58:41 -05:00
Jeff Washington (jwash) 3887169db0
lamports += to checked_add (#16841) 2021-04-27 09:56:18 -05:00
Jeff Washington (jwash) 47ca7063f2
rootstracker.remove returns previous state (#16831) 2021-04-27 09:15:44 -05:00
Jeff Washington (jwash) 3fdbaefaa6
tests: lamports -= to checked_sub (#16843) 2021-04-27 09:12:48 -05:00
Jeff Washington (jwash) 998cba74b5
AccountSharedData.executable() (#16835) 2021-04-27 09:12:17 -05:00
Jeff Washington (jwash) 81402ee7f5
pass &Pubkey through account storage, slot clean code to reduce copies (#16778)
* &Pubkey

* use trait to pass &Hash or Hash

* use impl Borrow<Hash> instead of trait

* remove old code line commented out
2021-04-27 09:10:06 -05:00
Jeff Washington (jwash) 603872685d
private AccountSharedData.owner (#16760)
* private AccountSharedData.owner

* fix perf test
2021-04-26 14:59:17 -05:00
Jeff Washington (jwash) f2ab0384e4
owner -> owner() (#16783) 2021-04-26 17:06:40 +00:00
Jeff Washington (jwash) aeff911c93
owner -> owner() (#16784) 2021-04-26 10:40:11 -05:00
Michael Vines d9dcd28d82 Ignore racy test_load_account_and_purge_race_without_retry 2021-04-23 23:17:56 +00:00
Jeff Washington (jwash) ca14c18998
owner -> owner() (#16782) 2021-04-23 22:49:47 +00:00
Jeff Washington (jwash) 1a4a7059af
owner -> owner() (#16785) 2021-04-23 15:59:13 -05:00
Jeff Washington (jwash) 48c07d32f0
WritableAccount.add/subtract_lamports (#16750)
* add/sub lamports

* make add/sub return Result

* sample replacements

* cleanup

* fix up a few tests as examples

* move enum, cleanup, impl from

* fmt

* cleanup

* add lamports.rs
2021-04-23 20:20:48 +00:00
Trent Nelson be29568318 runtime: checked math for Bank::withdraw 2021-04-23 20:13:35 +00:00
Jeff Washington (jwash) dcf2d84b24
refactor complicated 'len' (#16777) 2021-04-23 18:36:49 +00:00
Jeff Washington (jwash) 04e6aebf35
.owner = X -> .set_owner(X) (#16759) 2021-04-23 18:26:33 +00:00
Jeff Washington (jwash) fb0b76c1f3
account.owner = X -> account.set_owner(X) (#16754) 2021-04-23 12:34:22 -05:00
Jeff Washington (jwash) 7602911f9a
read_only_cache remove updates cache size (#16742) 2021-04-23 12:00:17 -05:00
Jeff Washington (jwash) 1cc9a1c0eb
log roots range metric (#16636)
* log roots range metric

* rename
2021-04-23 16:09:39 +00:00
Jeff Washington (jwash) 96d21335e1
.owner -> .owner() (#16758) 2021-04-23 09:35:55 -05:00
Jeff Washington (jwash) 5591cd6ef3
.owner -> .owner() (#16757)
* .owner -> .get_owner()

* use
2021-04-23 09:35:09 -05:00
Jeff Washington (jwash) 6c5b7ecf90
account.owner = X -> account.set_owner(X) (#16756) 2021-04-23 09:34:28 -05:00
Jeff Washington (jwash) 91be2903da
Reduce account index lookups during clean (#16689)
* reduce account index lookups in clean

* rename

* rename enum

* hold locks during removal from zero pubkey list

* merge with zero lamport fix

* tests
2021-04-23 09:33:14 -05:00
Jeff Washington (jwash) fc12841d95
Readonlyaccounts (#16743)
* lamports -> lamports()

* format
2021-04-22 20:04:55 +00:00
Jeff Washington (jwash) 739f0698ab
clean read_only_cache in the bg (#16722)
* clean read_only_cache in the bg

* reset lru_index

* cleanup

* cleanup

* drop for read only cache

* join bg thread

* reuse existing lru vec

* simplify reserve logic
2021-04-22 13:59:58 -05:00
Jeff Washington (jwash) 333998d008
.lamports = <number> -> .set_lamports(<number>) (#16746) 2021-04-22 18:56:47 +00:00
Jeff Washington (jwash) 8a6b80095e
Set lamports (#16747)
* lamports = -> set_lamports()

* .lamports = X -> .set_lamports(X)
2021-04-22 13:53:06 -05:00
Jeff Washington (jwash) 8d9d6b62d9
use cheaper account get (#16682) 2021-04-22 12:54:08 -05:00
Jeff Washington (jwash) 123e0bdba7
don't iterate slot_list if zero-lamport (#16678) 2021-04-21 20:38:48 -05:00
Jeff Washington (jwash) 69cbad0869
some ReadableAccount changes (#16688)
* some ReadableAccount changes

* deref
2021-04-21 12:20:37 -05:00
Jeff Washington (jwash) 189d2121e6
simplify do_shrink_slot_stores, delay/reduce account clone (#16691) 2021-04-21 11:17:38 -05:00
Jeff Washington (jwash) 9852572eb9
increase rolling slot width to 4M (#16583) 2021-04-20 21:18:15 -05:00
Jeff Washington (jwash) 03f7b251b8
ReadableAccount.data returns slice (#16686)
* ReadAbleAccount.data returns slice

* more slice fixup

* more slice

* slice
2021-04-20 16:41:16 -05:00
Jeff Washington (jwash) 36e11998c7
read only accounts cache uses accurate size representation (#16610)
* read ony accounts cache uses accurate size representation

* add comment and test
2021-04-20 09:58:01 -05:00
Jack May 01786f684e
Remove unwrap (#16652) 2021-04-19 20:17:56 -07:00
Jeff Washington (jwash) 4aa753ff01
rename threads: 15 char limit (#16625) 2021-04-19 12:16:58 -05:00
Alexander Meißner 9dfcb921cf
Refactoring: Move KeyedAccounts to InvokeContext (#15410)
Collects all parametric occurrences and the construction of keyed_accounts and puts them into InvokeContext.
2021-04-19 18:48:48 +02:00
Jeff Washington (jwash) 015bc034a5
improve failing assert (#16581) 2021-04-19 08:55:01 -05:00
Michael Vines a911ae00ba clippy 2021-04-18 20:55:02 -07:00
Trent Nelson 285f3c9d56 Feature-gate hash-based duplicate transaction check 2021-04-16 18:51:18 +00:00
carllin d747614b27
Account for possibility of cache flush in load() (#15454)
* Account for possibility of cache flush in load()

* More cleaning

* More cleaning

* Remove unused method and some comment cleaning

* Fix typo

* Make the detected impossible purge race panic()!

* Finally revert to original .expect()

* Fix typos...

* Add assertion for max_root for easier reasoning

* Reframe races with LoadHint as possible opt.

* Fix test

* Make race bug tests run longer for less flaky

* Delay the clone-in-lock slow path even for RPC

* Make get_account panic-free & add its onchain ver.

* Fix rebase conflicts...

* Clean up

* Clean up comment

* Revert fn name change

* Fix flaky test...

* fmt...

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-04-17 00:23:32 +09:00
Brooks Prumo 17aa45fad1 Remove old way of account hashing
Account data hashing used to use different ways of hashing on different
clusters.  That is no longer the case, but the old code still existed.
This commit removes that old, now used code.

**NOTE** The golden hash values in bank.rs needed to be updated.  Since
the original code that selected the hash algorithm used `if >` instead
of `if >=`, this meant that the genesis block's hash _always_ used the
old hashing method, which is no longer valid.

Validated by running `cargo test` successfully.
2021-04-13 14:42:21 -05:00
Justin Starry f641429056
Remove blake3 from bpf program dependencies (#16506) 2021-04-13 10:06:05 +00:00
Justin Starry 85eb37fab0
Merge pull request from GHSA-8v47-8c53-wwrc
* Track transaction check time separately from account loads

* banking packet process metrics

* Remove signature clone in status cache lookup

* Reduce allocations when converting packets to transactions

* Add blake3 hash of transaction messages in status cache

* Bug fixes

* fix tests and run fmt

* Address feedback

* fix simd tx entry verification

* Fix rebase

* Feedback

* clean up

* Add tests

* Remove feature switch and fall back to signature check

* Bump programs/bpf Cargo.lock

* clippy

* nudge benches

* Bump `BankSlotDelta` frozen ABI hash`

* Add blake3 to sdk/programs/Cargo.lock

* nudge bpf tests

* short circuit status cache checks

Co-authored-by: Trent Nelson <trent@solana.com>
2021-04-13 00:28:08 -06:00
Jack May fa83f3bd73
Return sysvars via syscalls (#16422) 2021-04-12 16:04:57 -07:00
Brooks Prumo 54ef065cc8 Use slot 1 for tests' sample store and accounts
Anticipating the removal of the old account hashing way, the Account DB
tests use slot 0 for the sample storages and accounts data.  Slot 0 is a
special case, and so to make future changes not break any tests, update
just the tests here now to use any other slot.

The golden hash values were obtained by running `cargo test` and copying
in the "actual" values from the test output into the `raw_expected`
vector of the test code.

Validated by running `cargo test` successfully.
2021-04-12 17:09:07 -05:00
Jeff Washington (jwash) 105a6bfb46
Replace RootsTracker HashSet (#16310)
* Replace RootsTracker HashSet

* use bitvec

* cleanup, add brenchmark test

* test cleanup

* add lots of tests

* get rid of demo

* change warp test constant

* get rid of unused function

* pr feedback

* reorder use

* rework get_all to remove range checks

* add tests, fix bugs
2021-04-12 12:11:33 -05:00
Jeff Washington (jwash) 7ba3e710d3
latest_slot uses > current_max (#16374) 2021-04-12 10:52:24 -05:00
Jeff Washington (jwash) 6930a77a0f
prepare replace Ancestors HashMap for performance (#16476) 2021-04-12 10:51:57 -05:00
carllin dc7030ffaa
Allow fork choice to support multiple versions of a slot (#16266) 2021-04-12 01:00:59 -07:00
Trent Nelson ac5462e7a2
Merge pull request from GHSA-fmvj-vqp5-qqh9
* Sanitize permissions

* Forbid creating directories under ledger/rocksdb/

* hardened_unpack: Disallow dirs under rocksdb/ in genesis

* hardened_unpack: expand valid genesis entry test coverage

* hardened_unpack: rework old-style bsd directory entry rejection

Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
2021-04-10 00:57:32 -06:00
François Garillot b08cff9e77
Simplify some pattern-matches (#16402)
When those match an exact combinator on Option / Result.

Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust).
2021-04-08 12:40:37 -06:00
steviez e69f37a464
Remove old function: account_balance_for_capitalization (#16383)
This function currently returns one of its' parameters and is thus useless.
2021-04-07 23:57:40 -05:00
Jack May 476fd40948
Add program-test tests to bpf programs (#16407) 2021-04-07 01:47:15 -07:00
Jack May 92f4018b07
Cleanup unsupported sysvars (#16390)
* Cleanup unsupported sysvars

* fix ser description
2021-04-06 00:08:03 -07:00
Jeff Washington (jwash) 95dc7b5449
reduce lock contention on latest_root (#16306) 2021-04-05 11:35:14 -05:00
Jeff Washington (jwash) 7a997759fa
reduce locking on get_rooted_entries (#16304) 2021-04-05 11:26:53 -05:00
Justin Starry 0596cf5405
Remove unprocessed transactions from log notifications (#16349) 2021-04-04 23:44:59 +08:00
Jeff Washington (jwash) 3996b699dc
read only account cache for executable accounts - improve replay (#16150)
* read only account cache

* tests

* clippy

* cleanup

* new file, add tests

* remove copy/paste code from test

* remove dead code

* all loads use cache

* remove stale comments

* add metrics logging for read only cache size

* report read only cache hits and misses

* consistency

* formatting

* rename, add comment

* u64

* better interaction with existing cache

* lru list saved between cleans
2021-04-01 07:16:34 -05:00
behzad nouri 3f63ed9a72
removes OrderedIterator and transaction batch iteration order (#16153)
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817

However load_and_execute_transactions is iterating over
  lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.

If iteratorion_order is [1, 2, 0], and i is 0, then:
  lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
  transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).

This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.
2021-03-31 23:59:19 +00:00
Jeff Washington (jwash) 416312b30b
update timings only on outer call (#16275)
* update timings only on outer call

* test & format
2021-03-31 18:02:59 -05:00
Jeff Washington (jwash) f374b35944
Hash stored accounts in bg (#16157)
* lazy calculate account hash

* push to bg thread

* remove deadlock

* logs

* format

* some cleanup on aisle 9

* format, fix up some metrics

* fix test, remove legacy function only there for tests

* cleanup

* remove unused store_hasher

* Switch to crossbeam

* clippy

* format

* use iter()

* rework from feedback

* hash_slot -> slot

* hash(cluster_type)

Co-authored-by: Carl Lin <carl@solana.com>
2021-03-31 15:39:34 -05:00
Jeff Washington (jwash) a5dcee254d
get rid of unnecessary data copy (#16214)
* get rid of unnecessary data copy

* preserve rent_epoch
2021-03-31 09:11:39 -05:00
sakridge 54c68ea83f
Drop write lock on sysvars (#15497)
* Drop write lock on sysvars

* adds env var for demoting sysvar write lock demotion

* moves demote logic to is_writable

* feature gates sysvar write lock demotion

* adds builtins to write lock demotion

* adds system program id to builtins

* adds Feature111...

* adds an abi-freeze test

* mvines set of builtin program keys

Co-authored-by: Michael Vines <mvines@gmail.com>

* update tests

* adds bpf loader keys

* Add test sysvar

* Plumb demote_sysvar to is_writable

* more plumbing of demote_sysvar_write_locks to is_writable

* patches test_program_bpf_instruction_introspection

* hard codes demote_sysvar_write_locks to false for serialization/encoding methods

* Revert "hard codes demote_sysvar_write_locks to false for serialization/encoding methods"

This reverts commit ae3e2d2e777437bddd753933097a210dcbc1b1fc.

* change the hardcoded ones to demote_sysvar_write_locks=true

* Use data_as_mut_slice

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-03-30 10:05:09 -07:00
sakridge 5e5b63712b
Status cache improvements (#16174) 2021-03-28 19:10:14 -07:00
Jeff Washington (jwash) 66c42f62d8
make AccountSharedData.data private to abstract storage (#16091)
* format

* 2 more tests

* use
2021-03-25 11:04:20 -05:00
Ryo Onodera 6d5c6c17c5
Simplify account.rent_epoch handling for sysvar rent (#16049)
* Add some code for special local testing

* Add comment to store_account_and_update_capitalization

* Simplify account.rent_epoch handling for sysvar rent

* Introduce *_for_test functions

* Add deprecation messages to existing api
2021-03-25 15:23:20 +09:00
sakridge 26af2d4c26
Up store limit for test_account_grow (#16099)
There can be more than 5 stores
2021-03-24 15:24:17 -07:00
Jeff Washington (jwash) 2d24d13046
AccountsSharedData: data copy on write (#15800)
* Arc<AccountData>

* try custom serializer

* adapt test from Behzad's change

* clippy

* simplify serialization

* remove abi example derive

* refactor 'take'

* remove serialization

* remove serialize calls

* remove account_data

* remove intos

* remove left over file
2021-03-23 15:19:31 -05:00
Jeff Washington (jwash) fde43a906d
revert test to use Account for serialization (#16063) 2021-03-22 22:48:09 +00:00
Jeff Washington (jwash) 4beb39f7a1
multi-pass bin scanning (#15377)
* multi-pass bin scanning

* pr feedback

* format

* fix typo

* adjust metrics for code changes

* merge errors
2021-03-18 10:32:07 -05:00
carllin f548a04fae
Allow unbounded wallclock processing time in tests (#15961) 2021-03-17 15:48:50 -07:00
DimAn a5d144b00f Revert to removing only tmp- 2021-03-17 10:31:01 -07:00
DimAn 0209d334bd Add option for separate snapshot location
(cherry picked from commit 6126878f509c69e23480a5ec22b3271e2b16e072)
2021-03-17 10:31:01 -07:00
Michael Vines a2eb655322 =1.7.0 2021-03-16 07:51:07 +00:00
Jeff Washington (jwash) efee8b62d7
a few missed set_data calls (#15846)
* a few missed set_data calls

* another set data call
2021-03-15 21:57:23 -05:00
Jeff Washington (jwash) e35d738906
rework BankClient.get_account_data (#15853) 2021-03-15 19:27:53 -05:00
carllin c1ba265dd9
Wallclock BankingStage Throttle (#15731) 2021-03-15 17:11:15 -07:00
Michael Vines 0c9ca5522c Bump version to v1.7.0 2021-03-13 09:01:21 +00:00
Justin Starry 918d04e3f0
Add more slot update notifications (#15734)
* Add more slot update notifications

* fix merge

* Address feedback and add integration test

* switch to datapoint

* remove unused shred method

* fix clippy

* new thread for rpc completed slots

* remove extra constant

* fixes

* rely on channel closing

* fix check
2021-03-12 21:44:06 +08:00
Ryo Onodera 4bbeb9c033
Remove old feature: simple_capitalization (#15763)
* Remove old feature: simple_capitalization

* Fix another failing test in core

* Finish up test cleanup

* Further clean up a bit
2021-03-12 11:12:40 +09:00
Jeff Washington (jwash) 952c3bcbb7
AccountSharedData construction (#15790) 2021-03-11 18:09:04 -06:00
Jeff Washington (jwash) 3419a5446e
AccountSharedData.set_data (#15781)
* account.set_data and resize_data

* remove data_resize
2021-03-11 16:40:45 -06:00
Michael Vines 1061d021c9 Unpack snapshot AppendVecs directly into account paths 2021-03-11 08:25:24 -08:00
Trent Nelson 24d18b3cf2 docs: add docs links for crates published to crates.io 2021-03-11 08:38:18 +00:00
Jack May 478518308d
Share RO and Executable accounts within invocations (#15799) 2021-03-10 23:04:00 -08:00
Michael Vines 65dd177601 AppendVec cleanup: remove dead code, rename non-idiomatic "new" function 2021-03-10 14:50:59 -08:00
Jeff Washington (jwash) 1135ffd595
mut data refs as slice (#15782) 2021-03-10 15:28:03 -06:00
Tyera Eulberg 9c1198c0c7
Improve load_largest_accounts more (#15785)
* Add load_largest_accounts bench

* Check lamports before address filter

* Use BinaryHeap, add Accounts test

* Use pubkey reference in the min-heap

Also, flatten code with early returns

Co-authored-by: Greg Fitzgerald <greg@solana.com>
2021-03-10 11:22:02 -07:00
Jack May 369e13b111
cleanup old runtime features (#15787) 2021-03-10 09:48:41 -08:00
Tyera Eulberg 3355efbb5a
Fix test to use AccountSharedData 2021-03-09 20:47:54 -07:00
Tyera Eulberg 5991cef5f5
Improve load_largest_accounts and add Bank test (#15775) 2021-03-10 00:23:41 +00:00
Jeff Washington (jwash) 52e54e1100
account.data -> data() (#15778) 2021-03-09 22:31:33 +00:00
Jeff Washington (jwash) 8a3135d17b
Account->AccountSharedData (#15691) 2021-03-09 15:06:07 -06:00
Tyera Eulberg e3e8179f2d
Sdk: start to appease clippy's integer arithmetic check (#15736)
* Move to module-level clippy allowance

* Fix stake_weighted_timestamp math

* Fix genesis_config

* Fix shred_version

* Fix hard_forks

* Fix process_instruction

* Add ArithmeticOverflow ix error

* Fix nonce_keyed_account

* Update BankSlotDelta frozen abi due to new ix error
2021-03-08 18:37:57 -07:00