95dc7b5449
reduce lock contention on latest_root ( #16306 )
2021-04-05 11:35:14 -05:00
7a997759fa
reduce locking on get_rooted_entries ( #16304 )
2021-04-05 11:26:53 -05:00
0596cf5405
Remove unprocessed transactions from log notifications ( #16349 )
2021-04-04 23:44:59 +08:00
3996b699dc
read only account cache for executable accounts - improve replay ( #16150 )
...
* read only account cache
* tests
* clippy
* cleanup
* new file, add tests
* remove copy/paste code from test
* remove dead code
* all loads use cache
* remove stale comments
* add metrics logging for read only cache size
* report read only cache hits and misses
* consistency
* formatting
* rename, add comment
* u64
* better interaction with existing cache
* lru list saved between cleans
2021-04-01 07:16:34 -05:00
3f63ed9a72
removes OrderedIterator and transaction batch iteration order ( #16153 )
...
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817
However load_and_execute_transactions is iterating over
lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.
If iteratorion_order is [1, 2, 0], and i is 0, then:
lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).
This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.
2021-03-31 23:59:19 +00:00
416312b30b
update timings only on outer call ( #16275 )
...
* update timings only on outer call
* test & format
2021-03-31 18:02:59 -05:00
f374b35944
Hash stored accounts in bg ( #16157 )
...
* lazy calculate account hash
* push to bg thread
* remove deadlock
* logs
* format
* some cleanup on aisle 9
* format, fix up some metrics
* fix test, remove legacy function only there for tests
* cleanup
* remove unused store_hasher
* Switch to crossbeam
* clippy
* format
* use iter()
* rework from feedback
* hash_slot -> slot
* hash(cluster_type)
Co-authored-by: Carl Lin <carl@solana.com>
2021-03-31 15:39:34 -05:00
a5dcee254d
get rid of unnecessary data copy ( #16214 )
...
* get rid of unnecessary data copy
* preserve rent_epoch
2021-03-31 09:11:39 -05:00
54c68ea83f
Drop write lock on sysvars ( #15497 )
...
* Drop write lock on sysvars
* adds env var for demoting sysvar write lock demotion
* moves demote logic to is_writable
* feature gates sysvar write lock demotion
* adds builtins to write lock demotion
* adds system program id to builtins
* adds Feature111...
* adds an abi-freeze test
* mvines set of builtin program keys
Co-authored-by: Michael Vines <mvines@gmail.com>
* update tests
* adds bpf loader keys
* Add test sysvar
* Plumb demote_sysvar to is_writable
* more plumbing of demote_sysvar_write_locks to is_writable
* patches test_program_bpf_instruction_introspection
* hard codes demote_sysvar_write_locks to false for serialization/encoding methods
* Revert "hard codes demote_sysvar_write_locks to false for serialization/encoding methods"
This reverts commit ae3e2d2e777437bddd753933097a210dcbc1b1fc.
* change the hardcoded ones to demote_sysvar_write_locks=true
* Use data_as_mut_slice
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-03-30 10:05:09 -07:00
5e5b63712b
Status cache improvements ( #16174 )
2021-03-28 19:10:14 -07:00
66c42f62d8
make AccountSharedData.data private to abstract storage ( #16091 )
...
* format
* 2 more tests
* use
2021-03-25 11:04:20 -05:00
6d5c6c17c5
Simplify account.rent_epoch handling for sysvar rent ( #16049 )
...
* Add some code for special local testing
* Add comment to store_account_and_update_capitalization
* Simplify account.rent_epoch handling for sysvar rent
* Introduce *_for_test functions
* Add deprecation messages to existing api
2021-03-25 15:23:20 +09:00
26af2d4c26
Up store limit for test_account_grow ( #16099 )
...
There can be more than 5 stores
2021-03-24 15:24:17 -07:00
2d24d13046
AccountsSharedData: data copy on write ( #15800 )
...
* Arc<AccountData>
* try custom serializer
* adapt test from Behzad's change
* clippy
* simplify serialization
* remove abi example derive
* refactor 'take'
* remove serialization
* remove serialize calls
* remove account_data
* remove intos
* remove left over file
2021-03-23 15:19:31 -05:00
fde43a906d
revert test to use Account for serialization ( #16063 )
2021-03-22 22:48:09 +00:00
4beb39f7a1
multi-pass bin scanning ( #15377 )
...
* multi-pass bin scanning
* pr feedback
* format
* fix typo
* adjust metrics for code changes
* merge errors
2021-03-18 10:32:07 -05:00
f548a04fae
Allow unbounded wallclock processing time in tests ( #15961 )
2021-03-17 15:48:50 -07:00
a5d144b00f
Revert to removing only tmp-
2021-03-17 10:31:01 -07:00
0209d334bd
Add option for separate snapshot location
...
(cherry picked from commit 6126878f509c69e23480a5ec22b3271e2b16e072)
2021-03-17 10:31:01 -07:00
a2eb655322
=1.7.0
2021-03-16 07:51:07 +00:00
efee8b62d7
a few missed set_data calls ( #15846 )
...
* a few missed set_data calls
* another set data call
2021-03-15 21:57:23 -05:00
e35d738906
rework BankClient.get_account_data ( #15853 )
2021-03-15 19:27:53 -05:00
c1ba265dd9
Wallclock BankingStage Throttle ( #15731 )
2021-03-15 17:11:15 -07:00
0c9ca5522c
Bump version to v1.7.0
2021-03-13 09:01:21 +00:00
918d04e3f0
Add more slot update notifications ( #15734 )
...
* Add more slot update notifications
* fix merge
* Address feedback and add integration test
* switch to datapoint
* remove unused shred method
* fix clippy
* new thread for rpc completed slots
* remove extra constant
* fixes
* rely on channel closing
* fix check
2021-03-12 21:44:06 +08:00
4bbeb9c033
Remove old feature: simple_capitalization ( #15763 )
...
* Remove old feature: simple_capitalization
* Fix another failing test in core
* Finish up test cleanup
* Further clean up a bit
2021-03-12 11:12:40 +09:00
952c3bcbb7
AccountSharedData construction ( #15790 )
2021-03-11 18:09:04 -06:00
3419a5446e
AccountSharedData.set_data ( #15781 )
...
* account.set_data and resize_data
* remove data_resize
2021-03-11 16:40:45 -06:00
1061d021c9
Unpack snapshot AppendVecs directly into account paths
2021-03-11 08:25:24 -08:00
24d18b3cf2
docs: add docs links for crates published to crates.io
2021-03-11 08:38:18 +00:00
478518308d
Share RO and Executable accounts within invocations ( #15799 )
2021-03-10 23:04:00 -08:00
65dd177601
AppendVec cleanup: remove dead code, rename non-idiomatic "new" function
2021-03-10 14:50:59 -08:00
1135ffd595
mut data refs as slice ( #15782 )
2021-03-10 15:28:03 -06:00
9c1198c0c7
Improve load_largest_accounts more ( #15785 )
...
* Add load_largest_accounts bench
* Check lamports before address filter
* Use BinaryHeap, add Accounts test
* Use pubkey reference in the min-heap
Also, flatten code with early returns
Co-authored-by: Greg Fitzgerald <greg@solana.com>
2021-03-10 11:22:02 -07:00
369e13b111
cleanup old runtime features ( #15787 )
2021-03-10 09:48:41 -08:00
3355efbb5a
Fix test to use AccountSharedData
2021-03-09 20:47:54 -07:00
5991cef5f5
Improve load_largest_accounts and add Bank test ( #15775 )
2021-03-10 00:23:41 +00:00
52e54e1100
account.data -> data() ( #15778 )
2021-03-09 22:31:33 +00:00
8a3135d17b
Account->AccountSharedData ( #15691 )
2021-03-09 15:06:07 -06:00
e3e8179f2d
Sdk: start to appease clippy's integer arithmetic check ( #15736 )
...
* Move to module-level clippy allowance
* Fix stake_weighted_timestamp math
* Fix genesis_config
* Fix shred_version
* Fix hard_forks
* Fix process_instruction
* Add ArithmeticOverflow ix error
* Fix nonce_keyed_account
* Update BankSlotDelta frozen abi due to new ix error
2021-03-08 18:37:57 -07:00
beac2d6d3f
Remove stale comment
2021-03-08 10:18:32 +09:00
8b0c6db871
Remove old feature: cumulative_rent_related_fixes ( #15754 )
2021-03-08 09:58:50 +09:00
c5371fdc85
More debug on shrink and flush ( #15518 )
2021-03-05 14:10:34 -08:00
13b9bb6790
refactor function to store into account after data is finished ( #15710 )
2021-03-05 11:02:35 -06:00
edd159e7d7
Fix typos ( #15721 )
2021-03-05 07:16:53 +00:00
872f7117c3
Use OrderedIterator to produce TransactionLogInfo ( #15712 )
...
* Add failing test
* Fix iteration_order issue with stored logs
2021-03-04 15:26:19 -07:00
1e2f5a5f55
Permit the snapshots/status_cache file to be sparse
2021-03-04 12:36:34 -08:00
b53cb8eb2d
Deprecate Instruction::new ( #15695 )
2021-03-04 05:46:48 +00:00
be35c1c1b7
add execute detail timings ( #15638 )
2021-03-03 17:07:45 -06:00
9bbe015a05
Document and clean up AppendVec ( #15640 )
...
* Document AppendVec
* Remove the almost-duplicate state in AppendVec
AppendVec was maintaining two offsets, `current_len` and
`append_offset`. Despite the different looking names, the two
values have the same meaning, but were updated at slightly different
times. When appending a batch of accounts, `current_len` updates would
be immediately available to other threads after each append, whereas
`append_offset` would only be updated after its mutex was unlocked.
`append_offset` is redundant. By removing it, we eliminate potential
bugs and no longer need to suppress clippy warnings.
* Remove get_mut() from AppendVec design
Only the offset into the AppendVec memory is thread-safe. The memory
itself is only threadsafe because it is append-only and is otherwise
unprotected. Adding get_mut() would only be safe if the memory was
protected by a ReadWrite lock.
2021-03-03 12:51:48 -08:00
a9c8dbfd0c
Add error reporting to system program ( #15644 )
2021-03-02 21:44:25 -08:00
a2c47db920
extract expected cluster type ( #15615 )
...
* extract expected cluster type
* fix clippy warning
* more clippy
2021-03-02 13:55:34 -06:00
43663b1750
Fix clippy warnings ( #15632 )
2021-03-02 08:58:14 -08:00
da47ab7e50
fix indentation format problem ( #15618 )
...
* fix indentation format problem
* fix indentation format problem
2021-03-02 10:46:00 -06:00
aad79d9d5c
Remove unused Accounts state ( #15611 )
...
* Remove dead state from Accounts
* Refactor to remove reference to self.slot
Account::default() has zero lamports, so would return None anyway
2021-03-02 08:28:25 -08:00
2463cc1e6a
Fix typos ( #15610 )
2021-03-02 06:36:49 -08:00
97eaf3c334
Fix finalize_dead_slot_removal() of cached slots decrementing refcount ( #15534 )
2021-02-26 17:49:37 -08:00
8399851d11
check program owners ( #15495 )
...
* check program owners
* BankSlotDelta should change because InstructionError variant added
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-02-26 22:21:34 +00:00
21b43009f6
Introduce ttl eviction for RecycleStore ( #15513 )
2021-02-25 17:27:27 +09:00
eddb7f98f5
Check vote account initialization ( #15503 )
...
* Check account data_len on Vote account init
* Check account data populated on update_cached_accounts
2021-02-24 10:00:48 -07:00
2f46da346d
gracefully handle vote account without authorized voter ( #15501 )
2021-02-24 00:00:24 -08:00
9a7cd8885d
Remove read only locks when they hit ref count zero, cleanup accounts locking ( #15449 )
2021-02-22 23:56:43 -08:00
5df36aec7d
Pacify clippy
2021-02-19 20:08:41 -08:00
5d31763a3b
Update ABI digest
2021-02-19 10:54:39 -08:00
02432a548f
Cleanup old features ( #15391 )
2021-02-18 09:56:11 -08:00
e3006e3b46
correctly reserve space for pre_accounts vector ( #15378 )
2021-02-18 09:37:24 -06:00
0f6f6080f3
sdk: Add Borsh support for types and utilities ( #15290 )
...
* sdk: Add Borsh to Pubkey
* Add serialization error for easier borsh integration
* Add Borsh usage to banks-client and sdk
* Rename SerializationError -> IOError
* Add new errors to proto
* Update Cargo lock
* Update Cargo.lock based on CI
* Clippy
* Update ABI on bank
* Address review feedback
* Update sanity program instruction count test
2021-02-18 11:14:56 +01:00
7f7370c306
Re-allow clippy::integer_arithmetic at crate-level
2021-02-17 13:55:08 -07:00
9ba69a7381
Adapt to fs_extra 1.2.0
2021-02-17 12:44:58 -08:00
7fee0bc69c
move hashing functionality to accounts_hash ( #15353 )
2021-02-17 00:29:50 +00:00
8367740ff9
compute merkle root on chunks of fanout^3 ( #15344 )
...
* compute merkle root on chunks of fanout^3
* improve test_accountsdb_compute_merkle_root_large
2021-02-16 17:03:35 -06:00
ba02452d75
add validator flag no-accounts-db-index-hashing ( #15350 )
...
* add validator flag no_accounts_db_index_hashing
* add validator flag no_accounts_db_index_hashing
2021-02-16 21:13:48 +00:00
d83aeebcca
change 2d cumulative calculation ( #15346 )
2021-02-16 14:24:45 -06:00
708b8380b7
assert 2 items in stores don't have same slot and version with different values ( #15250 )
...
* assert 2 items in stores don't have same slot and version with different values
* add message to assert
2021-02-16 18:12:59 +00:00
53959b4bbc
divide account scan into bins for more efficient downstream processing ( #15303 )
2021-02-15 21:27:39 -06:00
78a176fc00
rename CumulativeOffset in preparation of 2d support ( #15338 )
2021-02-15 20:57:05 -06:00
30f18319f2
Track RecycleStore basic stats with needed refactor ( #15291 )
...
* Track RecycleStore basic stats with needed refactor
* Fix another wrong metrics def
2021-02-15 16:09:17 +09:00
866ce236b1
add another hash check ( #15278 )
...
* add another hash check
* update comment
2021-02-12 18:35:11 -06:00
b8448f4189
merkle root code no longer adds lamports ( #15298 )
2021-02-12 17:30:14 -06:00
2e7aebf0bb
Rework spl_token_v2_self_transfer_fix to avoid any SPL Token downtime
2021-02-12 22:46:39 +00:00
8944efddf7
combine flatten and hash ( #15269 )
2021-02-12 11:35:54 -06:00
eced3776b0
add metrics to hash calculation and consolidate ( #15165 )
...
* add metrics to hash calculation
* add metrics to hash calculation
* create HashStats struct to consolidate hash timing
* formatting
2021-02-11 20:06:04 -06:00
eb54e544c2
add div_ceil ( #15267 )
2021-02-11 20:05:41 -06:00
59e55f0371
remove hash accumulation with lamports ( #15275 )
2021-02-11 20:05:22 -06:00
47c60f8e98
Clean up mainnet-beta inflation candidate features
2021-02-10 17:46:22 -08:00
e59a24d9f9
pass expected capitalization to hash calculation to improve assert msg ( #15191 )
...
* cleanup if
* pass expected capitalization to hash calculation to improve assert message
* fix bank function
* one more level
* calculate_accounts_hash_helper
* add slot to error message
* success
2021-02-10 14:38:00 -06:00
da6753b8c0
Warp timestamp and extend max-allowable-drift for accommodate slow blocks ( #15204 )
...
* Remove timestamp_correction feature gating
* Remove timestamp_bounding feature gating
* Remove unused deprecated ledger code
* Remove unused deprecated unbounded-timestamp code
* Enable independent adjustment of fast/slow timestamp bounding
* Update timestamp bounds to 25% fast, 80% slow; warp timestamp
* Update bank hash test
* Add PR number to feature
Co-authored-by: Michael Vines <mvines@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-09 15:49:00 -07:00
8424fe2c12
use index version of calculating hash ( #15189 )
...
* use index version of calculating hash
* invert const
* formatting
2021-02-09 10:37:50 -06:00
600cea274d
change store account scan to not use dashmap ( #15104 )
...
* change store account scan to not use dashmap
* add test_accountsdb_de_dup_accounts_from_stores
* add tests
* add test_accountsdb_flatten_hash_intermediate
* add tests
* add sort test
* add test
* clippy
* first_slice -> is_first_slice
* comment
* use partial_cmp
2021-02-05 21:16:30 -06:00
fbf9dc47e9
Threadpool2 ( #15151 )
...
* rework thread pool for hash calculation
* rename
2021-02-05 18:48:16 -06:00
fabecdc86c
use thread pool for non-index hash calculations ( #15149 )
2021-02-05 19:48:55 +00:00
6fd5ec0e4c
caches descendants in bank forks ( #15107 )
2021-02-05 18:00:45 +00:00
f85be6259b
sentinel value for zero lamport accounts in hash scanning ( #15097 )
...
* sentinel value for zero lamport accounts in hash scanning
* fix test
2021-02-05 11:23:48 -06:00
c69027e5aa
chore: bump serde from 1.0.118 to 1.0.122 ( #15126 )
...
* chore: bump serde from 1.0.118 to 1.0.122
* Update programs/bpf/Cargo.lock
2021-02-05 19:23:24 +09:00
7d9f5ad525
remove unused arg from function ( #15096 )
2021-02-04 20:46:30 -06:00
f0d58f5549
move timer end outside if ( #15087 )
2021-02-04 20:43:23 -06:00
e4d0d4bfae
Fix pubkey refcount for shrink + clean ( #14987 )
2021-02-04 12:44:19 -08:00
f49a70e626
borrow storages ( #15088 )
2021-02-04 11:29:03 -06:00
e5225b7e68
Add ref count from storage ( #15078 )
2021-02-04 09:11:05 -08:00
e7f53182e2
fix merge conflict ( #15085 )
2021-02-04 09:20:50 -06:00
Jeff Washington (jwash)
Justin Starry
behzad nouri
sakridge
Ryo Onodera
carllin
DimAn
Michael Vines
Trent Nelson
Jack May
Tyera Eulberg
Tyera Eulberg
Greg Fitzgerald
Jon Cinque
carllin