9b74988fc6
calculate_accounts_hash does not throttle threads on startup ( #17338 )
2021-05-20 10:25:54 -05:00
a9aa533684
replace Ancestors HashMap for performance ( #16477 )
...
* replace Ancestors HashMap for performance
* add ancestors module
2021-05-20 10:11:56 -05:00
a544010b03
don't log shrink metrics on first call ( #17328 )
...
* don't log shrink metrics on first call
* simplify logic
2021-05-20 00:15:35 +00:00
32ec8341f9
generate_index inserts ideal initial data ( #17247 )
...
* improve insert into map initially
* rework towards single code path
* rename
* update test
2021-05-19 16:21:24 -05:00
b5302e7619
add calc_stored_meta_us metric ( #17318 )
2021-05-19 12:29:41 -05:00
c20b27bc8f
shink all in parallel on startup ( #17308 )
2021-05-19 12:15:24 -05:00
ed9cbd50f0
move Ancestors to its own module ( #17316 )
2021-05-19 11:50:34 -05:00
0781fe1b4f
Upgrade Rust to 1.52.0 ( #17096 )
...
* Upgrade Rust to 1.52.0
update nightly_version to newly pushed docker image
fix clippy lint errors
1.52 comes with grcov 0.8.0, include this version to script
* upgrade to Rust 1.52.1
* disabling Serum from downstream projects until it is upgraded to Rust 1.52.1
2021-05-19 09:31:47 -05:00
4788976517
rework slot list update ( #17232 )
2021-05-19 08:48:46 -05:00
7fe24c455c
simplify generate_index ( #17292 )
2021-05-19 08:11:41 -05:00
827355a6b1
Create solana-rpc crate and move subscriptions ( #17320 )
...
* Move non_circulating_supply to runtime
* Add solana-rpc crate and move max_slots
* Move subscriptions to solana-rpc
* Single use statements
2021-05-19 00:54:28 -06:00
a3c0833a1c
tests: cleanup test ( #17305 )
...
* tests: cleanup test
* 2 more replacements
2021-05-18 18:08:37 -05:00
75335b4f58
datapoint for verify_snapshot_bank ( #17306 )
2021-05-18 12:14:43 -05:00
2175fc0625
calculate account infos before lock ( #17245 )
2021-05-18 08:06:19 -05:00
3a56a56d69
StoredMetaWriteVersion ( #17293 )
2021-05-17 23:58:36 +00:00
bcbe155575
Minor test cleanup and comments ( #17283 )
2021-05-18 07:22:43 +09:00
d461aba6a4
don't try to build secondary indexes if we don't need them ( #17246 )
2021-05-17 17:21:15 -05:00
9d6837c904
fix comment ( #17286 )
2021-05-17 14:06:25 -05:00
d1db5448b9
hold lock to speed up insert ( #17194 )
...
* hold lock to speed up insert
* add tests
2021-05-17 11:58:33 -05:00
3745e0babc
minor refactoring of AccountsIndex in preparation of bigger things ( #17237 )
2021-05-14 15:27:10 -05:00
fd88db7339
ancestors::new -> default ( #17195 )
2021-05-14 15:20:59 -05:00
c6f7867aa1
log mismatch ( #17209 )
2021-05-14 09:50:11 -05:00
76dfacca61
Add documentation for BankForks::prune_non_rooted() ( #17213 )
2021-05-14 07:47:48 -05:00
27004f1b76
Return error for excluded secondary-index keys ( #17193 )
...
* Add runtime helpers to check secondary indexes for key
* Add custom rpc error
* Check secondary-index key inclusion in rpc
* Clone complete AccountSecondaryIndexes into rpc to avoid bank query
2021-05-13 21:04:21 +00:00
3dbc7744ab
metrics for generating index time ( #17192 )
...
* metrics for generating index time
* update metrics to include scan time
2021-05-13 14:32:19 -05:00
239ab8799c
Remove bloat from secondary indexes ( #17048 )
2021-05-12 15:29:30 -07:00
9c42a89a43
Issue #17008 -- make snapshot archives to hold on to configurable. ( #17158 )
...
* purge_old_snapshot_archives is changed to take an extra argument 'maximum_snapshots_to_retain' to control the max number of latest snapshot archives to retain. Note the oldest snapshot is always retained as before and is not subjected to this new options.
* The validator and ledger-tool executables are modified with a CLI argument --maximum-snapshots-to-retain. And the options are propagated down the call chains. Their corresponding shell scripts were changed accordingly.
* SnapshotConfig is modified to have an extra field for the maximum_snapshots_to_retain
* Unit tests are developed to cover purge_old_snapshot_archives
2021-05-12 10:32:27 -07:00
7d96f78821
include/exclude keys on account secondary index ( #17110 )
...
* AccountSecondaryIndexes.include/exclude
* use normal scan if key is not indexed
* add a test to ask for a scan for an excluded secondary index
* fix up cli args
2021-05-11 22:06:22 +00:00
f39dda00e0
type AccountSecondaryIndexes = HashSet ( #17108 )
2021-05-10 14:22:48 +00:00
a6a1355b80
fix secondary index test ( #17111 )
2021-05-10 08:54:30 -05:00
c616c34825
Add comment for another unspoken eager collection subtlety ( #17137 )
2021-05-10 22:44:46 +09:00
225ec00ec4
add comment ( #17059 )
2021-05-06 15:17:41 -05:00
dc0429f5e6
add metric for assumption ( #17061 )
2021-05-06 15:04:13 -05:00
edc2ab3e48
Test account index and store alignment ( #17038 )
...
* Use ReclaimResult::Default() instead of building subtypes
* Add test to ensure account_db store and index are aligned
2021-05-06 14:13:44 -05:00
d19526e6c2
Dump rent_collector/inflation with ledger-tool cap ( #17069 )
2021-05-06 19:29:46 +09:00
ffbe8906ed
get root lock once ( #16829 )
2021-05-05 15:17:45 -05:00
e6f49a3e79
Clean unrooted unfrozen banks ( #16580 ) ( #17003 )
...
Cleanup pubkeys when an unrooted, unfrozen bank is dropped. This is a
continuation of PR #16911 .
2021-05-05 15:02:02 -05:00
ab7c96aa81
insert accounts in parallel when building initial index ( #17040 )
...
* insert accounts in parallel when building initial index
* rename nits from pr review
* rename nits from pr review
* rename nits from pr review
* rename nits from pr review
2021-05-05 17:08:45 +00:00
3e0fed48e7
Don't recognize temp snapshots as possible snapshots to open
2021-05-05 08:45:03 -07:00
5786be13a4
flatten_hash_intermediate sets capacity first ( #17013 )
...
* flatten_hash_intermediate sets capacity first
* use iterator instead of for
2021-05-05 09:07:05 -05:00
d94e4ef7c9
More docs for bank.rs ( #17006 )
2021-05-03 17:37:18 +00:00
541aa5ad85
tests: lamports -> lamports() ( #16982 )
2021-05-03 10:45:54 -05:00
744ac1535f
tests: lamports -> ReadableAccount ( #16983 )
2021-05-03 10:45:15 -05:00
1fe131a5bb
Readable account trait for stored account meta ( #16690 )
...
* ReadableAccount for StoredAccountMeta
* add test
2021-04-30 16:25:02 -05:00
1a9954f85b
bank deposit checked_add error ( #16917 )
...
* bank deposit checked_add error
* add id
* rename variables
* rename error and metric
2021-04-30 16:22:17 -05:00
01308cd890
distribute_rent_to_validators checked_add_lamports unwrap ( #16847 )
...
* distribute_rent_to_validators checked_add_lamports unwrap
* make rent disappear on add failure
* add pubkey to message
* update message text
* don't store account that we failed to transfer to
* format
2021-04-30 16:19:20 -05:00
763c04adf3
lamports = -> .set_lamports() ( #16980 )
2021-04-30 21:17:05 +00:00
17e6bd579f
Clean unrooted dropped banks ( #16580 ) ( #16911 )
...
In a scenario where a bank is unrooted and dropped, any keys that exist
_only_ in that bank are now cleaned up.
This work was originally based on PR #15106 .
2021-04-30 15:34:38 -05:00
6f88aeac56
tests: .lamports= -> .set_lamports() ( #16977 )
...
* .lamports= -> .set_lamports()
* another
* fix
* another
2021-04-30 18:44:15 +00:00
0af84bb91e
lamports= -> .set_lamports() ( #16978 )
2021-04-30 18:20:54 +00:00
c4943f3cb0
checked_sub_lamports ( #16932 )
2021-04-30 09:56:34 -05:00
63c56b57a9
checked_sub_lamports with unwrap ( #16922 )
...
* checked_sub_lamports with unwrap
* avoid unwrap
2021-04-29 15:04:28 -05:00
c9c94a7ef2
checked_sub_lamports ( #16923 )
2021-04-29 12:38:21 -05:00
ca7b36ad8f
lamports -> lamports() ( #16920 )
2021-04-29 10:44:46 -05:00
23d67e4ac7
stretchy roots tracker ( #16830 )
...
* stretchy roots tracker
* rename hash to hash_set in tests
* update comment
* try 2 widths in test
* bool iter
* add assert
* helper function for bitfield insert/remove
* introduce RollingBitFieldTester
* another bool iter replacement
* map cleanup
* map to cloned
2021-04-29 09:11:28 -05:00
a7070a5ca7
set_lamports() ( #16921 )
2021-04-28 21:30:06 +00:00
21c75d9d29
lamports = -> set_lamports ( #16919 )
2021-04-28 15:39:54 -05:00
f533d3be77
Write account path impl ReadableAccount ( #16779 )
2021-04-28 15:29:22 -05:00
9070191b8b
stats on clean for removing roots ( #16849 )
...
* stats on clean for removing roots
* rename
* accumulate and swap metrics
2021-04-28 13:24:01 -05:00
77f1ffd84b
Collect uncleaned pubkeys from all slots ( #16786 )
...
While working on another issue (#16580 ), the list of uncleaned pubkeys
returned from `remove_uncleaned_slots_and_collect_pubkeys_up_to_slot()`
did not include unrooted slots. This meant that during cleaning,
unrooted slots would not have their pubkeys cleaned up properly.
Now, return all uncleaned pubkeys, regardless if the slot is rooted or
not. Additionally, update the tests to have unrooted slots to ensure
this behavior.
This is part two of PR #16879 , and originally based on PR #15106 .
2021-04-28 10:57:42 -05:00
a7a671b3aa
AccountSharedData.set_executable() ( #16881 )
2021-04-28 14:07:43 +00:00
2021255f91
executable() ( #16880 )
2021-04-28 08:53:54 -05:00
da3342759b
private AccountSharedData.rent_epoch ( #16877 )
2021-04-28 08:52:20 -05:00
6381ee38eb
reclaims unref accounts from index ( #16838 )
2021-04-28 08:50:38 -05:00
1864bc2080
write Option<AccountSharedData> ( #16874 )
...
* write Option<&AccountSharedData>
* add comment
2021-04-28 08:47:26 -05:00
1eaff394da
Refactor `collect_uncleaned_pubkeys_to_slot()` ( #16879 )
...
* Refactor `collect_uncleaned_pubkeys_to_slot()`
While working on another issue (#16580 ), I came across
`collect_unclean_pubkeys_to_slot()` and had difficulty understanding it.
Since the function does a few logically separate things, I split the
function up. I also added documentation, removed an unused return value,
and renamed the functions to be more specific.
This commit is to split up an existing PR (#16786 ), where I had both this
aesthetic change _and_ a behavioral change.
2021-04-28 08:16:12 -05:00
1bd623cd15
private AccountSharedData.rent_epoch ( #16844 )
2021-04-27 13:51:13 -05:00
8f56c116d7
tests: lamports += to checked_add ( #16842 )
2021-04-27 10:23:07 -05:00
d533f77301
bank.withdraw uses checked_sub_lamports ( #16848 )
...
* bank.withdraw uses checked_sub_lamports
* retain previous codepath for error
* map_err from clippy
2021-04-27 09:58:41 -05:00
3887169db0
lamports += to checked_add ( #16841 )
2021-04-27 09:56:18 -05:00
47ca7063f2
rootstracker.remove returns previous state ( #16831 )
2021-04-27 09:15:44 -05:00
3fdbaefaa6
tests: lamports -= to checked_sub ( #16843 )
2021-04-27 09:12:48 -05:00
998cba74b5
AccountSharedData.executable() ( #16835 )
2021-04-27 09:12:17 -05:00
81402ee7f5
pass &Pubkey through account storage, slot clean code to reduce copies ( #16778 )
...
* &Pubkey
* use trait to pass &Hash or Hash
* use impl Borrow<Hash> instead of trait
* remove old code line commented out
2021-04-27 09:10:06 -05:00
603872685d
private AccountSharedData.owner ( #16760 )
...
* private AccountSharedData.owner
* fix perf test
2021-04-26 14:59:17 -05:00
f2ab0384e4
owner -> owner() ( #16783 )
2021-04-26 17:06:40 +00:00
aeff911c93
owner -> owner() ( #16784 )
2021-04-26 10:40:11 -05:00
d9dcd28d82
Ignore racy test_load_account_and_purge_race_without_retry
2021-04-23 23:17:56 +00:00
ca14c18998
owner -> owner() ( #16782 )
2021-04-23 22:49:47 +00:00
1a4a7059af
owner -> owner() ( #16785 )
2021-04-23 15:59:13 -05:00
48c07d32f0
WritableAccount.add/subtract_lamports ( #16750 )
...
* add/sub lamports
* make add/sub return Result
* sample replacements
* cleanup
* fix up a few tests as examples
* move enum, cleanup, impl from
* fmt
* cleanup
* add lamports.rs
2021-04-23 20:20:48 +00:00
be29568318
runtime: checked math for Bank::withdraw
2021-04-23 20:13:35 +00:00
dcf2d84b24
refactor complicated 'len' ( #16777 )
2021-04-23 18:36:49 +00:00
04e6aebf35
.owner = X -> .set_owner(X) ( #16759 )
2021-04-23 18:26:33 +00:00
fb0b76c1f3
account.owner = X -> account.set_owner(X) ( #16754 )
2021-04-23 12:34:22 -05:00
7602911f9a
read_only_cache remove updates cache size ( #16742 )
2021-04-23 12:00:17 -05:00
1cc9a1c0eb
log roots range metric ( #16636 )
...
* log roots range metric
* rename
2021-04-23 16:09:39 +00:00
96d21335e1
.owner -> .owner() ( #16758 )
2021-04-23 09:35:55 -05:00
5591cd6ef3
.owner -> .owner() ( #16757 )
...
* .owner -> .get_owner()
* use
2021-04-23 09:35:09 -05:00
6c5b7ecf90
account.owner = X -> account.set_owner(X) ( #16756 )
2021-04-23 09:34:28 -05:00
91be2903da
Reduce account index lookups during clean ( #16689 )
...
* reduce account index lookups in clean
* rename
* rename enum
* hold locks during removal from zero pubkey list
* merge with zero lamport fix
* tests
2021-04-23 09:33:14 -05:00
fc12841d95
Readonlyaccounts ( #16743 )
...
* lamports -> lamports()
* format
2021-04-22 20:04:55 +00:00
739f0698ab
clean read_only_cache in the bg ( #16722 )
...
* clean read_only_cache in the bg
* reset lru_index
* cleanup
* cleanup
* drop for read only cache
* join bg thread
* reuse existing lru vec
* simplify reserve logic
2021-04-22 13:59:58 -05:00
333998d008
.lamports = <number> -> .set_lamports(<number>) ( #16746 )
2021-04-22 18:56:47 +00:00
8a6b80095e
Set lamports ( #16747 )
...
* lamports = -> set_lamports()
* .lamports = X -> .set_lamports(X)
2021-04-22 13:53:06 -05:00
8d9d6b62d9
use cheaper account get ( #16682 )
2021-04-22 12:54:08 -05:00
123e0bdba7
don't iterate slot_list if zero-lamport ( #16678 )
2021-04-21 20:38:48 -05:00
69cbad0869
some ReadableAccount changes ( #16688 )
...
* some ReadableAccount changes
* deref
2021-04-21 12:20:37 -05:00
189d2121e6
simplify do_shrink_slot_stores, delay/reduce account clone ( #16691 )
2021-04-21 11:17:38 -05:00
9852572eb9
increase rolling slot width to 4M ( #16583 )
2021-04-20 21:18:15 -05:00
03f7b251b8
ReadableAccount.data returns slice ( #16686 )
...
* ReadAbleAccount.data returns slice
* more slice fixup
* more slice
* slice
2021-04-20 16:41:16 -05:00
36e11998c7
read only accounts cache uses accurate size representation ( #16610 )
...
* read ony accounts cache uses accurate size representation
* add comment and test
2021-04-20 09:58:01 -05:00
01786f684e
Remove unwrap ( #16652 )
2021-04-19 20:17:56 -07:00
4aa753ff01
rename threads: 15 char limit ( #16625 )
2021-04-19 12:16:58 -05:00
9dfcb921cf
Refactoring: Move KeyedAccounts to InvokeContext ( #15410 )
...
Collects all parametric occurrences and the construction of keyed_accounts and puts them into InvokeContext.
2021-04-19 18:48:48 +02:00
015bc034a5
improve failing assert ( #16581 )
2021-04-19 08:55:01 -05:00
a911ae00ba
clippy
2021-04-18 20:55:02 -07:00
285f3c9d56
Feature-gate hash-based duplicate transaction check
2021-04-16 18:51:18 +00:00
d747614b27
Account for possibility of cache flush in load() ( #15454 )
...
* Account for possibility of cache flush in load()
* More cleaning
* More cleaning
* Remove unused method and some comment cleaning
* Fix typo
* Make the detected impossible purge race panic()!
* Finally revert to original .expect()
* Fix typos...
* Add assertion for max_root for easier reasoning
* Reframe races with LoadHint as possible opt.
* Fix test
* Make race bug tests run longer for less flaky
* Delay the clone-in-lock slow path even for RPC
* Make get_account panic-free & add its onchain ver.
* Fix rebase conflicts...
* Clean up
* Clean up comment
* Revert fn name change
* Fix flaky test...
* fmt...
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-04-17 00:23:32 +09:00
17aa45fad1
Remove old way of account hashing
...
Account data hashing used to use different ways of hashing on different
clusters. That is no longer the case, but the old code still existed.
This commit removes that old, now used code.
**NOTE** The golden hash values in bank.rs needed to be updated. Since
the original code that selected the hash algorithm used `if >` instead
of `if >=`, this meant that the genesis block's hash _always_ used the
old hashing method, which is no longer valid.
Validated by running `cargo test` successfully.
2021-04-13 14:42:21 -05:00
85eb37fab0
Merge pull request from GHSA-8v47-8c53-wwrc
...
* Track transaction check time separately from account loads
* banking packet process metrics
* Remove signature clone in status cache lookup
* Reduce allocations when converting packets to transactions
* Add blake3 hash of transaction messages in status cache
* Bug fixes
* fix tests and run fmt
* Address feedback
* fix simd tx entry verification
* Fix rebase
* Feedback
* clean up
* Add tests
* Remove feature switch and fall back to signature check
* Bump programs/bpf Cargo.lock
* clippy
* nudge benches
* Bump `BankSlotDelta` frozen ABI hash`
* Add blake3 to sdk/programs/Cargo.lock
* nudge bpf tests
* short circuit status cache checks
Co-authored-by: Trent Nelson <trent@solana.com>
2021-04-13 00:28:08 -06:00
fa83f3bd73
Return sysvars via syscalls ( #16422 )
2021-04-12 16:04:57 -07:00
54ef065cc8
Use slot 1 for tests' sample store and accounts
...
Anticipating the removal of the old account hashing way, the Account DB
tests use slot 0 for the sample storages and accounts data. Slot 0 is a
special case, and so to make future changes not break any tests, update
just the tests here now to use any other slot.
The golden hash values were obtained by running `cargo test` and copying
in the "actual" values from the test output into the `raw_expected`
vector of the test code.
Validated by running `cargo test` successfully.
2021-04-12 17:09:07 -05:00
105a6bfb46
Replace RootsTracker HashSet ( #16310 )
...
* Replace RootsTracker HashSet
* use bitvec
* cleanup, add brenchmark test
* test cleanup
* add lots of tests
* get rid of demo
* change warp test constant
* get rid of unused function
* pr feedback
* reorder use
* rework get_all to remove range checks
* add tests, fix bugs
2021-04-12 12:11:33 -05:00
7ba3e710d3
latest_slot uses > current_max ( #16374 )
2021-04-12 10:52:24 -05:00
6930a77a0f
prepare replace Ancestors HashMap for performance ( #16476 )
2021-04-12 10:51:57 -05:00
dc7030ffaa
Allow fork choice to support multiple versions of a slot ( #16266 )
2021-04-12 01:00:59 -07:00
ac5462e7a2
Merge pull request from GHSA-fmvj-vqp5-qqh9
...
* Sanitize permissions
* Forbid creating directories under ledger/rocksdb/
* hardened_unpack: Disallow dirs under rocksdb/ in genesis
* hardened_unpack: expand valid genesis entry test coverage
* hardened_unpack: rework old-style bsd directory entry rejection
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
2021-04-10 00:57:32 -06:00
b08cff9e77
Simplify some pattern-matches ( #16402 )
...
When those match an exact combinator on Option / Result.
Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust ).
2021-04-08 12:40:37 -06:00
e69f37a464
Remove old function: account_balance_for_capitalization ( #16383 )
...
This function currently returns one of its' parameters and is thus useless.
2021-04-07 23:57:40 -05:00
476fd40948
Add program-test tests to bpf programs ( #16407 )
2021-04-07 01:47:15 -07:00
92f4018b07
Cleanup unsupported sysvars ( #16390 )
...
* Cleanup unsupported sysvars
* fix ser description
2021-04-06 00:08:03 -07:00
95dc7b5449
reduce lock contention on latest_root ( #16306 )
2021-04-05 11:35:14 -05:00
7a997759fa
reduce locking on get_rooted_entries ( #16304 )
2021-04-05 11:26:53 -05:00
0596cf5405
Remove unprocessed transactions from log notifications ( #16349 )
2021-04-04 23:44:59 +08:00
3996b699dc
read only account cache for executable accounts - improve replay ( #16150 )
...
* read only account cache
* tests
* clippy
* cleanup
* new file, add tests
* remove copy/paste code from test
* remove dead code
* all loads use cache
* remove stale comments
* add metrics logging for read only cache size
* report read only cache hits and misses
* consistency
* formatting
* rename, add comment
* u64
* better interaction with existing cache
* lru list saved between cleans
2021-04-01 07:16:34 -05:00
3f63ed9a72
removes OrderedIterator and transaction batch iteration order ( #16153 )
...
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817
However load_and_execute_transactions is iterating over
lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.
If iteratorion_order is [1, 2, 0], and i is 0, then:
lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).
This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.
2021-03-31 23:59:19 +00:00
416312b30b
update timings only on outer call ( #16275 )
...
* update timings only on outer call
* test & format
2021-03-31 18:02:59 -05:00
f374b35944
Hash stored accounts in bg ( #16157 )
...
* lazy calculate account hash
* push to bg thread
* remove deadlock
* logs
* format
* some cleanup on aisle 9
* format, fix up some metrics
* fix test, remove legacy function only there for tests
* cleanup
* remove unused store_hasher
* Switch to crossbeam
* clippy
* format
* use iter()
* rework from feedback
* hash_slot -> slot
* hash(cluster_type)
Co-authored-by: Carl Lin <carl@solana.com>
2021-03-31 15:39:34 -05:00
a5dcee254d
get rid of unnecessary data copy ( #16214 )
...
* get rid of unnecessary data copy
* preserve rent_epoch
2021-03-31 09:11:39 -05:00
54c68ea83f
Drop write lock on sysvars ( #15497 )
...
* Drop write lock on sysvars
* adds env var for demoting sysvar write lock demotion
* moves demote logic to is_writable
* feature gates sysvar write lock demotion
* adds builtins to write lock demotion
* adds system program id to builtins
* adds Feature111...
* adds an abi-freeze test
* mvines set of builtin program keys
Co-authored-by: Michael Vines <mvines@gmail.com>
* update tests
* adds bpf loader keys
* Add test sysvar
* Plumb demote_sysvar to is_writable
* more plumbing of demote_sysvar_write_locks to is_writable
* patches test_program_bpf_instruction_introspection
* hard codes demote_sysvar_write_locks to false for serialization/encoding methods
* Revert "hard codes demote_sysvar_write_locks to false for serialization/encoding methods"
This reverts commit ae3e2d2e777437bddd753933097a210dcbc1b1fc.
* change the hardcoded ones to demote_sysvar_write_locks=true
* Use data_as_mut_slice
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-03-30 10:05:09 -07:00
5e5b63712b
Status cache improvements ( #16174 )
2021-03-28 19:10:14 -07:00
66c42f62d8
make AccountSharedData.data private to abstract storage ( #16091 )
...
* format
* 2 more tests
* use
2021-03-25 11:04:20 -05:00
6d5c6c17c5
Simplify account.rent_epoch handling for sysvar rent ( #16049 )
...
* Add some code for special local testing
* Add comment to store_account_and_update_capitalization
* Simplify account.rent_epoch handling for sysvar rent
* Introduce *_for_test functions
* Add deprecation messages to existing api
2021-03-25 15:23:20 +09:00
26af2d4c26
Up store limit for test_account_grow ( #16099 )
...
There can be more than 5 stores
2021-03-24 15:24:17 -07:00
2d24d13046
AccountsSharedData: data copy on write ( #15800 )
...
* Arc<AccountData>
* try custom serializer
* adapt test from Behzad's change
* clippy
* simplify serialization
* remove abi example derive
* refactor 'take'
* remove serialization
* remove serialize calls
* remove account_data
* remove intos
* remove left over file
2021-03-23 15:19:31 -05:00
fde43a906d
revert test to use Account for serialization ( #16063 )
2021-03-22 22:48:09 +00:00
4beb39f7a1
multi-pass bin scanning ( #15377 )
...
* multi-pass bin scanning
* pr feedback
* format
* fix typo
* adjust metrics for code changes
* merge errors
2021-03-18 10:32:07 -05:00
f548a04fae
Allow unbounded wallclock processing time in tests ( #15961 )
2021-03-17 15:48:50 -07:00
a5d144b00f
Revert to removing only tmp-
2021-03-17 10:31:01 -07:00
0209d334bd
Add option for separate snapshot location
...
(cherry picked from commit 6126878f509c69e23480a5ec22b3271e2b16e072)
2021-03-17 10:31:01 -07:00
efee8b62d7
a few missed set_data calls ( #15846 )
...
* a few missed set_data calls
* another set data call
2021-03-15 21:57:23 -05:00
e35d738906
rework BankClient.get_account_data ( #15853 )
2021-03-15 19:27:53 -05:00
c1ba265dd9
Wallclock BankingStage Throttle ( #15731 )
2021-03-15 17:11:15 -07:00
918d04e3f0
Add more slot update notifications ( #15734 )
...
* Add more slot update notifications
* fix merge
* Address feedback and add integration test
* switch to datapoint
* remove unused shred method
* fix clippy
* new thread for rpc completed slots
* remove extra constant
* fixes
* rely on channel closing
* fix check
2021-03-12 21:44:06 +08:00
4bbeb9c033
Remove old feature: simple_capitalization ( #15763 )
...
* Remove old feature: simple_capitalization
* Fix another failing test in core
* Finish up test cleanup
* Further clean up a bit
2021-03-12 11:12:40 +09:00
952c3bcbb7
AccountSharedData construction ( #15790 )
2021-03-11 18:09:04 -06:00
3419a5446e
AccountSharedData.set_data ( #15781 )
...
* account.set_data and resize_data
* remove data_resize
2021-03-11 16:40:45 -06:00
1061d021c9
Unpack snapshot AppendVecs directly into account paths
2021-03-11 08:25:24 -08:00
478518308d
Share RO and Executable accounts within invocations ( #15799 )
2021-03-10 23:04:00 -08:00
65dd177601
AppendVec cleanup: remove dead code, rename non-idiomatic "new" function
2021-03-10 14:50:59 -08:00
1135ffd595
mut data refs as slice ( #15782 )
2021-03-10 15:28:03 -06:00
9c1198c0c7
Improve load_largest_accounts more ( #15785 )
...
* Add load_largest_accounts bench
* Check lamports before address filter
* Use BinaryHeap, add Accounts test
* Use pubkey reference in the min-heap
Also, flatten code with early returns
Co-authored-by: Greg Fitzgerald <greg@solana.com>
2021-03-10 11:22:02 -07:00
369e13b111
cleanup old runtime features ( #15787 )
2021-03-10 09:48:41 -08:00
3355efbb5a
Fix test to use AccountSharedData
2021-03-09 20:47:54 -07:00
5991cef5f5
Improve load_largest_accounts and add Bank test ( #15775 )
2021-03-10 00:23:41 +00:00
52e54e1100
account.data -> data() ( #15778 )
2021-03-09 22:31:33 +00:00
8a3135d17b
Account->AccountSharedData ( #15691 )
2021-03-09 15:06:07 -06:00
e3e8179f2d
Sdk: start to appease clippy's integer arithmetic check ( #15736 )
...
* Move to module-level clippy allowance
* Fix stake_weighted_timestamp math
* Fix genesis_config
* Fix shred_version
* Fix hard_forks
* Fix process_instruction
* Add ArithmeticOverflow ix error
* Fix nonce_keyed_account
* Update BankSlotDelta frozen abi due to new ix error
2021-03-08 18:37:57 -07:00
beac2d6d3f
Remove stale comment
2021-03-08 10:18:32 +09:00
8b0c6db871
Remove old feature: cumulative_rent_related_fixes ( #15754 )
2021-03-08 09:58:50 +09:00
c5371fdc85
More debug on shrink and flush ( #15518 )
2021-03-05 14:10:34 -08:00
13b9bb6790
refactor function to store into account after data is finished ( #15710 )
2021-03-05 11:02:35 -06:00
edd159e7d7
Fix typos ( #15721 )
2021-03-05 07:16:53 +00:00
872f7117c3
Use OrderedIterator to produce TransactionLogInfo ( #15712 )
...
* Add failing test
* Fix iteration_order issue with stored logs
2021-03-04 15:26:19 -07:00
1e2f5a5f55
Permit the snapshots/status_cache file to be sparse
2021-03-04 12:36:34 -08:00
b53cb8eb2d
Deprecate Instruction::new ( #15695 )
2021-03-04 05:46:48 +00:00
be35c1c1b7
add execute detail timings ( #15638 )
2021-03-03 17:07:45 -06:00
9bbe015a05
Document and clean up AppendVec ( #15640 )
...
* Document AppendVec
* Remove the almost-duplicate state in AppendVec
AppendVec was maintaining two offsets, `current_len` and
`append_offset`. Despite the different looking names, the two
values have the same meaning, but were updated at slightly different
times. When appending a batch of accounts, `current_len` updates would
be immediately available to other threads after each append, whereas
`append_offset` would only be updated after its mutex was unlocked.
`append_offset` is redundant. By removing it, we eliminate potential
bugs and no longer need to suppress clippy warnings.
* Remove get_mut() from AppendVec design
Only the offset into the AppendVec memory is thread-safe. The memory
itself is only threadsafe because it is append-only and is otherwise
unprotected. Adding get_mut() would only be safe if the memory was
protected by a ReadWrite lock.
2021-03-03 12:51:48 -08:00
a9c8dbfd0c
Add error reporting to system program ( #15644 )
2021-03-02 21:44:25 -08:00
a2c47db920
extract expected cluster type ( #15615 )
...
* extract expected cluster type
* fix clippy warning
* more clippy
2021-03-02 13:55:34 -06:00
43663b1750
Fix clippy warnings ( #15632 )
2021-03-02 08:58:14 -08:00
da47ab7e50
fix indentation format problem ( #15618 )
...
* fix indentation format problem
* fix indentation format problem
2021-03-02 10:46:00 -06:00
aad79d9d5c
Remove unused Accounts state ( #15611 )
...
* Remove dead state from Accounts
* Refactor to remove reference to self.slot
Account::default() has zero lamports, so would return None anyway
2021-03-02 08:28:25 -08:00
2463cc1e6a
Fix typos ( #15610 )
2021-03-02 06:36:49 -08:00
97eaf3c334
Fix finalize_dead_slot_removal() of cached slots decrementing refcount ( #15534 )
2021-02-26 17:49:37 -08:00
8399851d11
check program owners ( #15495 )
...
* check program owners
* BankSlotDelta should change because InstructionError variant added
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-02-26 22:21:34 +00:00
21b43009f6
Introduce ttl eviction for RecycleStore ( #15513 )
2021-02-25 17:27:27 +09:00
eddb7f98f5
Check vote account initialization ( #15503 )
...
* Check account data_len on Vote account init
* Check account data populated on update_cached_accounts
2021-02-24 10:00:48 -07:00
2f46da346d
gracefully handle vote account without authorized voter ( #15501 )
2021-02-24 00:00:24 -08:00
9a7cd8885d
Remove read only locks when they hit ref count zero, cleanup accounts locking ( #15449 )
2021-02-22 23:56:43 -08:00
5df36aec7d
Pacify clippy
2021-02-19 20:08:41 -08:00
5d31763a3b
Update ABI digest
2021-02-19 10:54:39 -08:00
02432a548f
Cleanup old features ( #15391 )
2021-02-18 09:56:11 -08:00
e3006e3b46
correctly reserve space for pre_accounts vector ( #15378 )
2021-02-18 09:37:24 -06:00
0f6f6080f3
sdk: Add Borsh support for types and utilities ( #15290 )
...
* sdk: Add Borsh to Pubkey
* Add serialization error for easier borsh integration
* Add Borsh usage to banks-client and sdk
* Rename SerializationError -> IOError
* Add new errors to proto
* Update Cargo lock
* Update Cargo.lock based on CI
* Clippy
* Update ABI on bank
* Address review feedback
* Update sanity program instruction count test
2021-02-18 11:14:56 +01:00
7f7370c306
Re-allow clippy::integer_arithmetic at crate-level
2021-02-17 13:55:08 -07:00
9ba69a7381
Adapt to fs_extra 1.2.0
2021-02-17 12:44:58 -08:00
7fee0bc69c
move hashing functionality to accounts_hash ( #15353 )
2021-02-17 00:29:50 +00:00
8367740ff9
compute merkle root on chunks of fanout^3 ( #15344 )
...
* compute merkle root on chunks of fanout^3
* improve test_accountsdb_compute_merkle_root_large
2021-02-16 17:03:35 -06:00
ba02452d75
add validator flag no-accounts-db-index-hashing ( #15350 )
...
* add validator flag no_accounts_db_index_hashing
* add validator flag no_accounts_db_index_hashing
2021-02-16 21:13:48 +00:00
d83aeebcca
change 2d cumulative calculation ( #15346 )
2021-02-16 14:24:45 -06:00
708b8380b7
assert 2 items in stores don't have same slot and version with different values ( #15250 )
...
* assert 2 items in stores don't have same slot and version with different values
* add message to assert
2021-02-16 18:12:59 +00:00
53959b4bbc
divide account scan into bins for more efficient downstream processing ( #15303 )
2021-02-15 21:27:39 -06:00
78a176fc00
rename CumulativeOffset in preparation of 2d support ( #15338 )
2021-02-15 20:57:05 -06:00
30f18319f2
Track RecycleStore basic stats with needed refactor ( #15291 )
...
* Track RecycleStore basic stats with needed refactor
* Fix another wrong metrics def
2021-02-15 16:09:17 +09:00
866ce236b1
add another hash check ( #15278 )
...
* add another hash check
* update comment
2021-02-12 18:35:11 -06:00
b8448f4189
merkle root code no longer adds lamports ( #15298 )
2021-02-12 17:30:14 -06:00
2e7aebf0bb
Rework spl_token_v2_self_transfer_fix to avoid any SPL Token downtime
2021-02-12 22:46:39 +00:00
8944efddf7
combine flatten and hash ( #15269 )
2021-02-12 11:35:54 -06:00
eced3776b0
add metrics to hash calculation and consolidate ( #15165 )
...
* add metrics to hash calculation
* add metrics to hash calculation
* create HashStats struct to consolidate hash timing
* formatting
2021-02-11 20:06:04 -06:00
eb54e544c2
add div_ceil ( #15267 )
2021-02-11 20:05:41 -06:00
59e55f0371
remove hash accumulation with lamports ( #15275 )
2021-02-11 20:05:22 -06:00
47c60f8e98
Clean up mainnet-beta inflation candidate features
2021-02-10 17:46:22 -08:00
e59a24d9f9
pass expected capitalization to hash calculation to improve assert msg ( #15191 )
...
* cleanup if
* pass expected capitalization to hash calculation to improve assert message
* fix bank function
* one more level
* calculate_accounts_hash_helper
* add slot to error message
* success
2021-02-10 14:38:00 -06:00
da6753b8c0
Warp timestamp and extend max-allowable-drift for accommodate slow blocks ( #15204 )
...
* Remove timestamp_correction feature gating
* Remove timestamp_bounding feature gating
* Remove unused deprecated ledger code
* Remove unused deprecated unbounded-timestamp code
* Enable independent adjustment of fast/slow timestamp bounding
* Update timestamp bounds to 25% fast, 80% slow; warp timestamp
* Update bank hash test
* Add PR number to feature
Co-authored-by: Michael Vines <mvines@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-09 15:49:00 -07:00
8424fe2c12
use index version of calculating hash ( #15189 )
...
* use index version of calculating hash
* invert const
* formatting
2021-02-09 10:37:50 -06:00
600cea274d
change store account scan to not use dashmap ( #15104 )
...
* change store account scan to not use dashmap
* add test_accountsdb_de_dup_accounts_from_stores
* add tests
* add test_accountsdb_flatten_hash_intermediate
* add tests
* add sort test
* add test
* clippy
* first_slice -> is_first_slice
* comment
* use partial_cmp
2021-02-05 21:16:30 -06:00
fbf9dc47e9
Threadpool2 ( #15151 )
...
* rework thread pool for hash calculation
* rename
2021-02-05 18:48:16 -06:00
fabecdc86c
use thread pool for non-index hash calculations ( #15149 )
2021-02-05 19:48:55 +00:00
6fd5ec0e4c
caches descendants in bank forks ( #15107 )
2021-02-05 18:00:45 +00:00
f85be6259b
sentinel value for zero lamport accounts in hash scanning ( #15097 )
...
* sentinel value for zero lamport accounts in hash scanning
* fix test
2021-02-05 11:23:48 -06:00
7d9f5ad525
remove unused arg from function ( #15096 )
2021-02-04 20:46:30 -06:00
f0d58f5549
move timer end outside if ( #15087 )
2021-02-04 20:43:23 -06:00
e4d0d4bfae
Fix pubkey refcount for shrink + clean ( #14987 )
2021-02-04 12:44:19 -08:00
f49a70e626
borrow storages ( #15088 )
2021-02-04 11:29:03 -06:00
e5225b7e68
Add ref count from storage ( #15078 )
2021-02-04 09:11:05 -08:00
e7f53182e2
fix merge conflict ( #15085 )
2021-02-04 09:20:50 -06:00
600ff0d915
calculate hash from store instead of index ( #15034 )
...
* calculate hash from store instead of index
* restore update hash in abs
2021-02-04 09:00:33 -06:00
2970b59853
Don't load all accounts into memory for capitalization check ( #14957 )
...
* Don't load all accounts into memory for capitalization check
2021-02-03 15:00:42 -08:00
8e93a784f3
program-test: Add ability to warp to the future ( #14998 )
...
* program-test: Add ability to warp to the future
* Make `start_local_server` take by value
* Remove clear_invoke_context
2021-02-03 23:31:36 +01:00
2b9a0311d3
Don't reset accounts if the remove_account comes from a clean ( #15022 )
...
Store may be in-use with a snapshot creation, so don't disturb
it's state.
2021-02-03 13:06:28 -08:00
37aac5a12d
Cleanup v1 shrink path ( #15009 )
...
move legacy functions to another impl block
2021-02-02 13:29:46 -08:00
22800054e6
streamline calculate_accounts_hash ( #14980 )
2021-02-02 10:58:19 -06:00
31168fe343
Disable AppendVec warn! for now ( #14996 )
...
* Disable AppendVed warn! for now
* Fix version...
* Update append_vec.rs
2021-02-02 14:52:30 +00:00
1df93fa2be
chore: bump serde from 1.0.112 to 1.0.118 ( #14828 )
...
* chore: bump serde from 1.0.112 to 1.0.122
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.112 to 1.0.122.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.122 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Update frozen_abi digest following serde update
* Revert "chore: bump serde from 1.0.112 to 1.0.122"
This reverts commit a3ef4442a4c985144ae2bd7ceaf8899a7ab8d7c0.
* Revert "[auto-commit] Update all Cargo lock files"
This reverts commit c41c3b005fb1ccade55155302c52cd5736c4b55f.
* chore: bump serde from 1.0.112 to 1.0.118
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.112 to 1.0.118.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.118 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Remove serum-dex pinning
* blind commit!
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-02-02 23:28:16 +09:00
73d9186502
More rich runtime logging ( #14938 )
2021-02-01 11:40:49 -08:00
63c44bd690
Use helper for count() in accountsDB ( #14953 )
2021-01-31 09:32:02 -08:00
07cef5a557
Buffer authority must match upgrade authority for deploys and upgrades ( #14923 )
2021-01-29 12:43:42 -08:00
65315fa4c2
Use already-generated key set to populate dirty keys for clean ( #14905 )
...
Don't need to scan the stores again when we already found the key
set of updates per slot. Just insert it earlier.
2021-01-29 11:44:43 -08:00
85b5dbead6
Add feature for pending SPL Token self-transfer fix
2021-01-29 04:19:33 +00:00
0b1015f7d3
Richer runtime failure logging ( #14875 )
2021-01-28 10:04:54 -08:00
a53b8558cd
Only mmap file from snapshot once ( #14815 )
2021-01-28 08:15:33 -08:00
63429507b2
Add syscall feature activation test ( #14890 )
2021-01-27 17:21:25 -08:00
72f10f5f29
Aggregate purge and shrink metrics ( #14763 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-27 01:39:47 -08:00
daddcd361a
Snapshots missing slots from accounts cache clean optimization ( #14852 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-26 20:08:55 -08:00
ffa5c7dcc8
Deprecate commitment variants ( #14797 )
...
* Deprecate commitment variants
* Add new CommitmentConfig builders
* Add helpers to avoid allowing deprecated variants
* Remove deprecated transaction-status code
* Include new commitment variants in runtime commitment; allow deprecated as long as old variants persist
* Remove deprecated banks code
* Remove deprecated variants in core; allow deprecated in rpc/rpc-subscriptions for now
* Heavier hand with rpc/rpc-subscription commitment
* Remove deprecated variants from local-cluster
* Remove deprecated variants from various tools
* Remove deprecated variants from validator
* Update docs
* Remove deprecated client code
* Add new variants to cli; remove deprecated variants as possible
* Don't send new commitment variants to old clusters
* Retain deprecated method in test_validator_saves_tower
* Fix clippy matches! suggestion for BPF solana-sdk legacy compile test
* Refactor node version check to handle commitment variants and transaction encoding
* Hide deprecated variants from cli help
* Add cli App comments
2021-01-26 19:23:07 +00:00
2b50433099
Remove legacy_stake program
2021-01-25 19:34:40 -08:00
1b85114a9c
remove legacy merkle root ( #14772 )
...
* remove legacy merkle root
f78197a
* clippy
* compile error
* borrow error
* derministic results
* clippy
* borrow
2021-01-25 15:26:08 -06:00
5700e835ef
Refactors tuple of TransactionLoadResult into a struct. ( #14773 )
2021-01-25 10:35:08 +01:00
0d32a0e0f4
Partial clean ( #14800 )
...
* Revert "Revert "Partial accounts clean (#14652 )" (#14777 )"
This reverts commit ad2e10e17b
2021-01-24 09:50:19 -08:00
cbffab7850
Upgrade to Rust v1.49.0
2021-01-23 19:16:36 -08:00
96b27ca0cb
Remove zero_lamport_pubkeys reference ( #14808 )
2021-01-23 18:31:55 +00:00
424bb797a6
Speed up generate_index ( #14792 )
2021-01-23 08:05:05 -08:00
c77461e428
Remove unnecesary flushes in previous roots ( #14596 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2021-01-23 04:02:44 -08:00
bf1943e489
Add solana-test-validator --warp-slot argument
2021-01-22 21:17:02 -08:00
dd5a2ef05f
Add convenience function to create a snapshot archive out of any Bank
2021-01-22 21:17:02 -08:00
ad2e10e17b
Revert "Partial accounts clean ( #14652 )" ( #14777 )
...
This reverts commit ca7914aa3d
2021-01-22 18:05:49 -08:00
77572a7c53
Track account writable deescalation ( #14626 )
2021-01-22 15:28:01 -08:00
18bd0c9a5b
speed up merkle calculation ( #14710 )
2021-01-22 09:21:50 -06:00
Jeff Washington (jwash)
Tao Zhu
Tyera Eulberg
Ryo Onodera
Brooks Prumo
carllin
Lijun Wang
steviez
Stephen Akridge
Michael Vines
Trent Nelson
Jack May
Alexander Meißner
Justin Starry
François Garillot
steviez
behzad nouri
DimAn
Tyera Eulberg
Greg Fitzgerald
Jon Cinque
carllin
dependabot[bot]